Say “Aloha”: A closer look at Facebook’s voice ambitions

Facebook has been a bit slow to adopt the voice computing revolution. It has no voice assistant, its smart speaker is still in development, and some apps like Instagram aren’t full equipped for audio communication. But much of that is set to change judging by experiments discovered in Facebook’s code, plus new patent filings.

Developing voice functionality could give people more ways to use Facebook in their home or on the go. Its forthcoming Portal smart speaker is reportedly designed for easy video chatting with distant family, including seniors and kids that might have trouble with phones. Improved transcription and speech-to-text-to-speech features could connect Messenger users across input mediums and keep them on the chat app rather than straying back to SMS.

But Facebook’s voice could be drowned out by the din of the crowd if it doesn’t get moving soon. All the major mobile hardware and operating system makers now have their own voice assistants like Siri, Alexa, Google Assistant, and Samsung Bixby, as well as their own smart speakers. In Q2 2018, Canalys estimates that Google shipped 5.4 million Homes, and Amazon shipped 4.1 million Echoes. Apple’s HomePod is off to a slow start with less than 6 percent of the market, behind Alibaba’s smart speaker according to Strategy Analytics. Facebook’s spotty record around privacy might deflect potential customers to its competitors.

Given Facebook is late to the game, it will need to arrive with powerful utility that solves real problems. Here’s a look at Facebook’s newest developments in the voice space, and how its past experiments lay the groundwork for its next big push.

Aloha Voice

Facebook is developing its own speech recognition feature under the name Aloha for both the Facebook and Messenger apps, as well as external hardware — likely the video chat smart speaker it’s developing. Code inside the Facebook and Messenger Android apps dug up by frequent TechCrunch tipster and mobile researcher Jane Manchun Wong gives the first look at a prototype for the Aloha user interface.

Labeled “Aloha Voice Testing”, as a user speaks while in a message thread, a horizontal blue bar expands and contracts to visualize the volume of speech while recognizing and transcribing into text. The code describes the feature as having connections with external WiFi or Bluetooth devices. It’s possible that the software will run on both Facebook’s hardware and software, similar to Google Assistant that runs both on phones and Google Home speakers.

Facebook declined to comment on the video, with its spokesperson Ha Thai telling me “We test stuff all the time – nothing to share today but my team will be in touch in a few weeks about hardware news coming from the AR/VR org.” It unclear if that hardware news will focus on voice and Aloha or portal, or if it’s merely related to Facebook’s Oculus Connect 5 conference on September 25th.

A source previously told me that years ago, Facebook was interested in developing its own speech recognition software designed specifically to accurately transcribe how friends talk to each other. These speech patterns are often more casual, colloquial, rapid, and full of slang than the way we formally address computerized assistants like Amazon Alexa or Google Home.

Wong also found the Aloha logo buried in Facebook’s code, which features volcano imagery. I can confirm that I’ve seen a Facebook Aloha Setup chatbot with a similar logo on the phones of Facebook employees.

If Facebook can figure this out, it could offer its own transcription features in Messenger and elsewhere on the site so users could communicate across mediums. It could potentially let you dictate comments or messages to friends while you have your hands full or can’t look at your screen. The recipient could then read the text instead of having to listen to it like a voice message. The feature could also be used to power voice navigation of Facebook’s apps for better hands-free usage.

Speaker And Camera Patents

Facebook awarded patent for speaker

Facebook’s video chat smart speaker was reportedly codenamed Aloha originally but later renamed Portal, Alex Heath of Business Insider and now Cheddar first reported in August 2017. The $499 competitor to the Amazon Echo Show was initially set to launch at Facebook’s F8 in May, but Bloomberg reported it was pushed back amid concerns that it would exacerbate the privacy scandal ignited by Cambridge Analytica.

A new patent filing reveals Facebook was considering building a smart speaker as early as December 26th, 2016 when it filed a patent for a cube-shaped device. The patent diagrams an “ornamental design for a speaker device” invented by Baback Elmieh, Alexandre Jais, and John Proksch-Whaley. Facebook had acquired Elmieh’s startup Nascent Objects in September of that year and he’s now a technical project lead at Facebook’s secretive Building 8 hardware lab.

The startup had been building modular hardware, and earlier this year he was awarded patents for work at Facebook on several modular cameras. The speaker and camera technology Facebook has been developing could potentially evolve into what’s in its video chat speaker.

The fact that Facebook has been exploring speaker technology for so long and that the lead on these patents is still running a secret project in Building 8 strengthens the case that Facebook has big plans for the voice space.

Patents awarded to Facebook show designs for a camera (left) and video camera (right)

Instagram Voice Messaging

And finally, Instagram is getting deeper into the voice game too. A screenshot generated from the code of Instagram’s Android app by Wong reveals the development of a  voice clip messaging feature heading to Instagram Direct. This would allow you to speak into Instagram and send the audio clips similar to a walkie-talkie, or the voice messaging feature Facebook Messenger added back in 2013.

You can see the voice button in the message composer at the bottom of the screen, and the code explains that to “Voice message, press and hold to record”. The prototype follows the recent launch of video chat in Instagram Direct, another feature TechCrunch broke the news on thanks to Wong’s research. An Instagram spokesperson declined to comment, as is typical when features are spotted in its code but aren’t publicly testing yet, saying “unfortunately nothing more to share on this right now.”

The Long Road To Voicebook

Facebook has long tinkered in the voice space. In 2015, it acquired a natural language processing startup Wit.ai that ran a developer platform for building speech interfaces, though it later rolled Wit.ai into Messenger’s platform team to focus on chatbots. Facebook also began testing automatically transcribing Messenger voice clips into text in 2015 in what was likely the groundwork for the Aloha feature seen above. The company also revealed its M personal assistant that could accomplish tasks for users, but it was only rolled out to a very limited user base and later turned off.

The next year, Facebook’s head of Messenger David Marcus claimed at TechCrunch Disrupt that voice “is not something we’re actively working on right now,” but added that “at some point it’s pretty obvious that as we develop more and more capabilities and interactions inside of Messenger, we’ll start working on voice exchanges and interfaces.” However, a source had told me Facebook’s secretive Language Technology Group was already exploring voice opportunities. Facebook also began testing its Live Audio feature for users who want to just broadcast sound and not video.

By 2017, Facebook was offering automatic captioning for Pages’ videos, and was developing a voice search feature. And this year, Facebook began trying voice clips as status updates and Stories for users around the world who might have trouble typing in their native tongue. But executives haven’t spoken much about the voice initiatives.

The most detailed comments we have come from Facebook’s head of design Luke Woods at TechCrunch Disrupt 2017 where he described voice search saying it was, “very promising. There are lots of exciting things happening…. I love to be able to talk to the car to navigate to a particular place. That’s one of many potential use cases.” It’s also one that voice transcription could aid.

It’s still unclear exactly what Facebook’s Aloha will become. It could be a defacto operating system or voice interface and transcription feature for Facebook’s smart speaker and apps. It could become a more full-fledged voice assistant like M but with audio. Or perhaps it could become Facebook’s bridge to other voice ecosystems, serving as Facebook’s Alexa Skill or Google Assistant Action.

When I asked Woods “How would Facebook on Alexa work?”, he said with a smile “That’s a very interesting question! No comment.”

Facebook assigns you a fake news flagging trustworthiness score

A new way to attack Facebook is to fraudulently report a news story as false in hopes of reducing its visibility, either because someone wants to censor it or just doesn’t agree with it. Sometimes known as “brigading”, a concerted effort by trolls to flag a piece of content can reduce its visibility. Facebook now sends stories reported as false to third-party fact checkers, and these purposefully inaccurate reports can clog the already-overcrowded queues that fact checkers struggle to worth through.

That’s why Facebook gives users a trustworthiness score ranging from 0 to1 dependon the reliability of their flags of false news, the Washington Post reports. If they flag something as false news but fact checkers verify it as true, that could hurt their score and reduce how heavily Facebook factors in their future flagging. If users consistently report false news that’s indeed proven to be false, their score improves and Facebook will trust their future flagging more.

Facebook’s News Feed product manager Tessa Lyons confirmed the scoring system exists. There’s currently no way to see your own or someone else’s trustworthiness score. And other signals are used to compute the score as well, though Facebook won’t reveal them for fear of trolls gaming the system.

Friend Ranking Scores

This isn’t the only way Facebook ranks users, though. It assigns you a shifting score of affinity towards each of your friends that determines how frequently you see them in the News Feed. This “friend ranking” score is essentially a measure of graph distance from you to someone else.

If you like a ton of someone’s posts, get tagged in photos with them, search for them, view their profile, communicate with them, have lots of mutual friends, are in the same Groups, and have similar biographical characteristics like location and age, your score towards them is lower and you’ll see more of them in feed. However, they have a different score for you depending on their behavior, so constantly viewing someone else’s profile won’t make you show up in their feed more if they don’t reciprocate the interest.

I first reported on these friend scores almost exactly seven years ago, and you can still view them for yourself using this browser bookmarklet built by Jeremy Keeshin. Visit this site, drag the “Facebook Friends Rankings” link into your desktop browser’s bookmark bar, open Facebook while logged in, and tap the bookmarklet to reveal the Friend Ranking scores of your friends. It snoops the Facebook JavaScript to pull out the scores. The people you see at the top are who you’re closest to.

Facebook clearly The need for this score highlights the difficulties of Facebook’s battle against fake news. Between subjectivity and purposeful trolling, there’s a lot noise coming in with the signal about what should be removed. Anyone saying Facebook should have easily solved the fake news problems is likely underappreciating the nuance required and the intelligent human adversaries Facebook must defeat.

Facebook has a huge array of signals it can use to calculate Friend Rankings or trustworthiness scores. The question will be whether it can intelligently sort those signals to make coherent inferences about what to show us and when to believe us.

Tinder’s latest feature, Tinder U, is only for college students

Tinder is today rolling out what may be one of its smartest additions yet with the launch of Tinder U, a feature designed specifically for Tinder users in college. Once enabled, students with a .edu email address will be able to register with their school, then swipe on students who also attend their school or others nearby. Beyond limiting potential matches to other students, the overall Tinder experience is unchanged.

Students will still be able to view each others’ profiles, swipe right and left to match or pass, message mutual matches, use Super Likes, and more.

To use Tinder U, students will first have to be geolocated on campus and log in to the Tinder app using their .edu email address. They’ll then have to check their inbox for the verification email and tap the button to confirm their account.

After completing this process, users will be in the Tinder U experience the next time they launch the app.

Here, students will see their school’s logo appear at the top of the screen, and individual profile photos will have flair on the bottom left to indicate the user’s school. Tinder U doesn’t prevent users from swiping off campus, however – using a toggle button at the top of the screen, users can choose to swipe by location instead, or by Tinder Picks by toggling over to the diamond icon (if available).

Tinder U makes sense for the company, whose user base already skews younger – it has said before that half its user base is between 18 and 24, for example. And dating apps’ usage, in general, among this age group has roughly tripped from 10% in 2013 to 27% by 2016, according to Pew Research. And of course, there’s the fact that Tinder itself got its start on college campuses – a market that’s young, single, and more willing to adopt mobile dating apps than other, older demographics.

Tinder says the new feature is launching initially on iOS devices at 4-year, accredited, not-for-profit schools in the U.S. that deliver courses in a traditional face-to-face learning format – meaning, no online universities or virtual schools will be supported. The company didn’t provide a timeframe for the Android release.