## Why Use The Smart Glasses Manager?
### Developers
Building apps that run on a smart phone and stream data to smart glasses is hard, but it’s how today’s lightweight smart glasses work. We handle the connection, UI, data streaming, and transcription, allowing you to rapidly develop smart glasses applications that run on any pair of smart glasses.
### Industry
Your smart glasses won’t see massive consumer adoption if third party developers can’t make awesome apps for your hardware. The SmartGlassesManager is a middleware which already supports many apps and makes it easy for developers to build more. That means, if you add support to your hardware for a single application – the SmartGlassesManager – your glasses will instantly support a plethora of consumer-facing applications.
## Features
1. Connect smart phone to smart glasses, auto-reconnect
- Wifi, Bluetooth, Android, MCU glasses all supported
2. Receive audio + sensors from glasses
3. Transcribe audio
4. Abstracted interface to show info on the glasses
## Fork of the Wearable Intelligence System
This repo is a fork of the [Wearable Intelligence System](https://github.com/emexlabs/WearableIntelligenceSystem). The Wearable Intelligence System was started at [Emex Labs](https://emexwearables.com) by [Cayden Pierce](https://caydenpierce.com/). This repo has a lot of the history cleaned to make it easier to manage, see the WIS repo for full history.
## Install / Use
You will need two pieces of hardware to run the system:
1. ASP - Android Smart Phone running Android 12+
2. A pair of smart glasses:
- Vuzix Z100 / Ultralite OEM Reference Platform
- Vuzix Shield
- Activelook Engo 1 or Activelook Engo 2
- Inmo Air 1 or Inmo Air 2
- Vuzix Blade 2
- TCL RayNeo X2
### Install
1. On your Android smart phone, flash the ASP app from Android Studio, the releases page, or from Google Play.
- Play Store: Coming soon
- [Github latest release](https://github.com/TeamOpenSmartGlasses/SmartGlassesManager/releases)
2. If using Android Smart Glasses, then on your smart glasses, download or flash the ASG client app:
- [Github latest release](https://github.com/TeamOpenSmartGlasses/SmartGlassesManager/releases)
3.
* Launch the "Smart Glasses Manager" app on your smart phone
* Accept permissions.
* If using Android Smart Glasses -> Tap "Start Wifi Hotspot", turn on (configure password if necessary) your wifi hotspot, then go "Back" to return
- Connect smart glasses WiFi to the smart phone WiFi hotspot
- Enable mobile data (or wifi sharing) on Android smart phone
- Start ASG application on smart glasses
- The phone connection icon will be green if the glasses are connected to your phone. If you speak, you'll see a live transcript on the smart glasses screen.
* If using Bluetooth Smart Glasses -> Turn on Bluetooth on phone.
3. Tap "Connect Smart Glasses" and choose your glasses.
4. Setup complete.
### Normal Use
Here's how to launch the system after you've already done the initial setup above:
1. Launch "Smart Glasses Manager" app on smart phone
2. Enable mobile hotspot on smart phone with the "Start WiFi Hotspot" button
3. Connect Android smart glasses to Android smart phone WiFi hotspot.
4. Launch "WIS" app on smart glasses.
5. Verify system is running by the "Smart Glasses Conection Indicator" icon turning white on the smart glasses HUD.
## Documentation / Developers
(WIP)
Here are the docs: https://github.com/TeamOpenSmartGlasses/SmartGlassesManager/wiki
## Authors
The system is fully open source and built by [TeamOpenSmartGlasses](https://teamopensmartglasses.com). We're a team building open source smart glasses technology to enhance user's intelligence in daily life. Join us: https://discord.gg/5ukNvkEAqT
The TeamOpenSmartGlasses members who are contributing to this project include:
- [Cayden Pierce](https://caydenpierce.com)
- [Alex Israelov](http://www.alexisraelov.com/)
We are building a community that is building cognitive augmentation technologies together.
================================================
FILE: SGM_android/.gitignore
================================================
*.iml
.gradle
/local.properties
/.idea/caches
/.idea/libraries
/.idea/modules.xml
/.idea/workspace.xml
/.idea/navEditor.xml
/.idea/assetWizardSettings.xml
.DS_Store
/build
/captures
.externalNativeBuild
.cxx
local.properties
================================================
FILE: SGM_android/README.md
================================================
### Special instructions if you need to mess with the OGG streaming CPP JNI.
0. Run Linux (as you should be).
1. Install Java 17.
2. Ensure Java 17 is default Java (can set with `sudo update-java-alternatives`).
3. Run `chmod 777 ./gradle/` and `chmod 777 ./gradle/`.
4. Set your ANDROID_SDK_PATH WITH `export $ANDROID_SDK_PATH=`.
5. Go into android folder and run `bash build_all.sh` to build everything.
6. If you get gradle version issues, install gradle 8.0.2: https://linuxhint.com/installing_gradle_ubuntu/ (follow the instructions, but replace 7.4.2 with 8.0.2).
7. Subsequent builds, you can just run `./gradlew assembleDebug --stacktrace` to build the APK.
8. Now that CPP/JNI is built, you can use Android Studio to edit app and install APK on phone (located in app/build/outputs/debug/). If you change anything related to CPP/JNI code, you may have to rebuild.
================================================
FILE: SGM_android/SmartGlassesManager/.gitignore
================================================
/build
================================================
FILE: SGM_android/SmartGlassesManager/CMakeLists.txt
================================================
cmake_minimum_required(VERSION 3.22.1)
project("some-name-needed-here")
add_subdirectory(src/main/cpp)
add_subdirectory(src/main/cpp/jni)
================================================
FILE: SGM_android/SmartGlassesManager/build.gradle
================================================
plugins {
id 'com.android.library'
// id 'kotlin-android'
id 'com.google.protobuf' version '0.9.2'
}
ext {
kotlin_version = '1.6.21'
roomVersion = '2.2.3'
archLifecycleVersion = '2.2.0-rc03'
coreTestingVersion = '2.1.0'
kotlinVersion = '1.6.21'
}
android {
namespace "com.teamopensmartglasses.smartglassesmanager"
compileSdk 33
// aaptOptions {
// noCompress "tflite"
// }
packagingOptions {
exclude 'META-INF/proguard/androidx-annotations.pro'
resources {
excludes += '/META-INF/{AL2.0,LGPL2.1,DEPENDENCIES}'
excludes += '/META-INF/INDEX.LIST'
}
}
defaultConfig {
minSdk 30
targetSdk 32
versionCode 3
versionName "0.1"
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
// ndk {
// abiFilters 'armeabi-v7a', 'arm64-v8a'
// }
}
externalNativeBuild {
cmake {
'-DANDROID_STL=c++_static'
}
}
// buildTypes {
// release {
// minifyEnabled false
// proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
// }
//// debug {
//// applicationIdSuffix ".debug" //so we can have both release and debug on the same phone
//// }
// }
// compileOptions {
// sourceCompatibility JavaVersion.VERSION_1_8
// targetCompatibility JavaVersion.VERSION_1_8
// }
//
// sourceSets {
// main {
// assets.srcDirs += "$buildDir/generated/assets"
// java.srcDirs += 'src/main/java'
// proto.srcDirs += 'src/main/proto'
// }
// }
//
// externalNativeBuild {
// cmake {
// path file('CMakeLists.txt')
// version '3.22.1'
// }
// }
}
dependencies {
implementation fileTree(dir: 'libs', include: ['*.jar'])
implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlinVersion"
implementation 'androidx.navigation:navigation-fragment:2.2.0'
implementation 'androidx.navigation:navigation-ui:2.2.0'
testImplementation 'junit:junit:4.+'
androidTestImplementation 'androidx.test.ext:junit:1.1.3'
implementation 'com.google.protobuf:protobuf-java:3.21.12'
implementation 'com.google.protobuf:protobuf-java-util:3.21.12'
implementation 'com.google.api.grpc:grpc-google-cloud-speech-v1p1beta1:2.6.0'
implementation "androidx.concurrent:concurrent-futures:1.0.0-alpha03"
implementation "androidx.lifecycle:lifecycle-common:2.2.0"
implementation 'androidx.preference:preference:1.2.0'
implementation 'joda-time:joda-time:2.12.2'
implementation "androidx.annotation:annotation:1.1.0"
implementation "androidx.appcompat:appcompat:1.1.0"
implementation "androidx.constraintlayout:constraintlayout:2.1.0"
implementation "androidx.core:core:1.1.0-rc03"
"androidx.legacy:legacy-support-v4:1.0.0"
implementation 'io.grpc:grpc-okhttp:1.53.0'
implementation "androidx.recyclerview:recyclerview:1.1.0-beta02"
implementation "com.github.bumptech.glide:glide:4.11.0"
implementation "com.google.android.material:material:1.6.0-alpha01"
implementation "com.google.code.findbugs:jsr305:3.0.2"
implementation "com.google.flogger:flogger-system-backend:0.3.1"
implementation "com.google.flogger:flogger:0.3.1"
implementation "com.google.guava:listenablefuture:1.0"
implementation "junit:junit:4.12"
implementation "org.hamcrest:hamcrest-library:1.3"
implementation 'com.squareup.picasso:picasso:2.8'
implementation "org.java-websocket:Java-WebSocket:1.5.2"
implementation "io.reactivex.rxjava3:rxandroid:3.0.0"
implementation "io.reactivex.rxjava3:rxjava:3.0.0"
implementation "com.google.android.gms:play-services-tasks:18.0.0"
implementation "com.google.android.gms:play-services-location:18.0.0"
implementation "com.google.code.gson:gson:2.8.6"
//ROOM
// Room components
implementation "androidx.room:room-runtime:$roomVersion"
annotationProcessor "androidx.room:room-compiler:$roomVersion"
androidTestImplementation "androidx.room:room-testing:$roomVersion"
// Lifecycle components
implementation "androidx.lifecycle:lifecycle-extensions:$archLifecycleVersion"
annotationProcessor "androidx.lifecycle:lifecycle-compiler:$archLifecycleVersion"
implementation "com.google.guava:guava:28.1-android"
//VOSK ASR
implementation 'net.java.dev.jna:jna:5.8.0@aar'
implementation "com.alphacephei:vosk-android:0.3.34@aar"
//java.util.time backport
implementation "org.threeten:threetenbp:1.5.2"
//http(s) lib
implementation "com.android.volley:volley:1.2.1"
//EventBus, greenrobot
implementation("org.greenrobot:eventbus:3.3.1")
//activelook SDK
//implementation 'com.github.activelook:android-sdk:v4.4.0'
//Json (de/)serialization
implementation 'com.google.code.gson:gson:2.10.1'
//for VAD (using Silero DNN)
implementation 'com.github.gkonovalov:android-vad:2.0.1'
implementation 'com.microsoft.onnxruntime:onnxruntime-android:1.15.0'
//Vuzix Ultralite OEM Platform / Vuzix Z100
implementation 'com.vuzix:ultralite-sdk-android:1.4.1'
//Azure ASR
implementation "com.microsoft.cognitiveservices.speech:client-sdk:1.38.0"
implementation project(path: ':AugmentOSLib')
}
//VOSK ASR prep models - need one for each language
tasks.register('genUUID_en') {
def uuid = UUID.randomUUID().toString()
def odir = file("$buildDir/generated/assets/model-en-us")
def ofile = file("$odir/uuid")
doLast {
mkdir odir
ofile.text = uuid
}
}
tasks.register('genUUID_fr') {
def uuid = UUID.randomUUID().toString()
def odir = file("$buildDir/generated/assets/model-fr-small")
def ofile = file("$odir/uuid")
doLast {
mkdir odir
ofile.text = uuid
}
}
preBuild.dependsOn(genUUID_en)
preBuild.dependsOn(genUUID_fr)
protobuf {
protoc {
// use this compiler, from the maven repo (instead of a local file, for instance)
artifact = 'com.google.protobuf:protoc:3.21.12'
}
generateProtoTasks {
all().configureEach { task ->
// remove the default java output
task.builtins {
remove java
}
// add the java output with lite runtime
task.builtins {
java {
option 'lite'
}
}
}
}
}
================================================
FILE: SGM_android/SmartGlassesManager/proguard-rules.pro
================================================
# Add project specific ProGuard rules here.
# You can control the set of applied configuration files using the
# proguardFiles setting in build.gradle.
#
# For more details, see
# http://developer.android.com/guide/developing/tools/proguard.html
# If your project uses WebView with JS, uncomment the following
# and specify the fully qualified class name to the JavaScript interface
# class:
#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
# public *;
#}
# Uncomment this to preserve the line number information for
# debugging stack traces.
#-keepattributes SourceFile,LineNumberTable
# If you keep the line number information, uncomment this to
# hide the original source file name.
#-renamesourcefileattribute SourceFile
#don't leave logs in release build:
-dontskipnonpubliclibraryclasses
-dontobfuscate
-forceprocessing
-optimizationpasses 5
-keep class * extends android.app.Activity
-assumenosideeffects class android.util.Log {
public static *** d(...);
public static *** v(...);
}
================================================
FILE: SGM_android/SmartGlassesManager/src/androidTest/java/com/teamopensmartglasses/smartglassesmanager/ExampleInstrumentedTest.java
================================================
package com.teamopensmartglasses.smartglassesmanager;
import android.content.Context;
import androidx.test.platform.app.InstrumentationRegistry;
import androidx.test.ext.junit.runners.AndroidJUnit4;
import org.junit.Test;
import org.junit.runner.RunWith;
import static org.junit.Assert.*;
/**
* Instrumented test, which will execute on an Android device.
*
* @see Testing documentation
*/
@RunWith(AndroidJUnit4.class)
public class ExampleInstrumentedTest {
@Test
public void useAppContext() {
// Context of the app under test.
Context appContext = InstrumentationRegistry.getInstrumentation().getTargetContext();
assertEquals("com.wearableintelligencesystem.androidsmartphone", appContext.getPackageName());
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/AndroidManifest.xml
================================================
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/CMakeLists.txt
================================================
# Builds the ogg and opus encoder library which used third party module.
cmake_minimum_required(VERSION 3.22.1)
# set related path of third party libraries.
set(third_party_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../../third_party)
# libogg library refer from https://github.com/xiph/ogg.
set(libogg_INCLUDE ${third_party_DIR}/libogg/include)
set(libogg_LIB ${third_party_DIR}/libogg/lib/${ANDROID_ABI})
add_library(lib_ogg STATIC IMPORTED)
set_target_properties(lib_ogg PROPERTIES IMPORTED_LOCATION ${libogg_LIB}/libogg.a)
# libopus library refer from https://github.com/xiph/opus.
set(libopus_INCLUDE ${third_party_DIR}/libopus/include)
set(libopus_LIB ${third_party_DIR}/libopus/lib/${ANDROID_ABI})
add_library(lib_opus STATIC IMPORTED)
set_target_properties(lib_opus PROPERTIES IMPORTED_LOCATION ${libopus_LIB}/libopus.a)
# opus-tools https://github.com/xiph/opus-tools.
set(opus_tools_INCLUDE ${third_party_DIR}/opus_tools/include)
set(opus_tools_LIB ${third_party_DIR}/opus_tools/lib/${ANDROID_ABI})
add_library(lib_opus_header STATIC IMPORTED)
set_target_properties(lib_opus_header PROPERTIES IMPORTED_LOCATION
${opus_tools_LIB}/libopus_header.a)
# Build ogg_opus_encoder_tool shared lib.
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11")
add_library(ogg_opus_encoder_tool SHARED ogg_opus_encoder.cc)
target_include_directories(ogg_opus_encoder_tool PRIVATE ${libogg_INCLUDE})
target_link_libraries(ogg_opus_encoder_tool lib_opus lib_ogg lib_opus_header)
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/jni/CMakeLists.txt
================================================
#build for ogg_opus_encoder shared library
cmake_minimum_required(VERSION 3.22.1)
add_library(ogg_opus_encoder SHARED ogg_opus_encoder.cc ../ogg_opus_encoder.cc)
# Include libraries needed for ogg_opus_encoder
target_link_libraries(ogg_opus_encoder ogg_opus_encoder_tool)
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/jni/ogg_opus_encoder.cc
================================================
/*
* Copyright 2019 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "ogg_opus_encoder.h"
#include
#include
#include
#include "../ogg_opus_encoder.h"
namespace {
using audio_util::OggOpusEncoder;
OggOpusEncoder* GetInstanceOrDie(jlong ptr) {
assert(ptr);
return reinterpret_cast(ptr);
}
bool VerifyInitialized(const std::string& function_name, jlong instance_ptr) {
if (instance_ptr == 0) {
fprintf(stderr, "%s called prior to allocate() or after destroy()!\n",
function_name.c_str());
return false;
}
OggOpusEncoder* instance = GetInstanceOrDie(instance_ptr);
if (instance == nullptr) {
fprintf(stderr, "%s called prior to init()!\n", function_name.c_str());
return false;
}
return true;
}
JNIEXPORT jbyteArray convertToByteArray(const std::vector& data,
JNIEnv* env) {
jbyteArray byteArray = env->NewByteArray(data.size());
env->SetByteArrayRegion(byteArray, 0, data.size(),
reinterpret_cast(data.data()));
return byteArray;
}
} // namespace
JNIEXPORT jlong JNICALL JNI_METHOD(init)(JNIEnv* env, jobject instance,
jint num_channels,
jint bitrate_bits_per_second,
jint sample_rate_hz,
jboolean use_vbr) {
constexpr bool low_latency_mode = true;
return reinterpret_cast(new OggOpusEncoder(
num_channels, sample_rate_hz, bitrate_bits_per_second, use_vbr,
low_latency_mode));
}
JNIEXPORT jbyteArray JNICALL
JNI_METHOD(processAudioBytes)(JNIEnv* env, jobject instance, jlong instance_ptr,
jbyteArray samples, jint offset, jint length) {
if (!VerifyInitialized("processAudioBytes", instance_ptr)) {
return convertToByteArray(std::vector(0), env);
}
jsize array_length_bytes = length;
if (!array_length_bytes) {
fprintf(stdout, "Found empty array\n");
}
assert(array_length_bytes % 2 == 0 && "int16 formatted stream missing bytes!");
std::vector pcm(array_length_bytes / 2);
env->GetByteArrayRegion(samples, offset, array_length_bytes,
reinterpret_cast(pcm.data()));
if (env->ExceptionOccurred()) {
fprintf(stderr, "Exception occurred in java Environment object\n");
}
return convertToByteArray(GetInstanceOrDie(instance_ptr)->Process(pcm), env);
}
JNIEXPORT jbyteArray JNICALL JNI_METHOD(flush)(JNIEnv* env, jobject instance,
jlong instance_ptr) {
return convertToByteArray(GetInstanceOrDie(instance_ptr)->Flush(), env);
}
JNIEXPORT void JNICALL JNI_METHOD(free)(JNIEnv* env, jobject instance,
jlong instance_ptr) {
delete reinterpret_cast(instance_ptr);
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/jni/ogg_opus_encoder.h
================================================
/*
* Copyright 2019 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef AUDIO_UTIL_JNI_OGG_OPUS_ENCODER_H_
#define AUDIO_UTIL_JNI_OGG_OPUS_ENCODER_H_
#include
// https://stackoverflow.com/questions/25363027/jni-getmethodid-not-working-for-constructor-of-inner-class
#define JNI_METHOD(fn) \
Java_com_teamopensmartglasses_smartglassesmanager_speechrecognition_google_asr_asrhelpers_StreamingAudioEncoder_00024OggOpusEncoder_##fn // NOLINT
extern "C" {
// Create opus encoder instance. The pointer is returned in a
// jlong. Remember to call destroy with the returned value when you're done.
JNIEXPORT jlong JNICALL JNI_METHOD(init)(JNIEnv* env,
jobject instance,
jint num_channels,
jint bitrate_bits_per_second,
jint sample_rate_hz,
jboolean use_vbr);
// samples must be an even number of bytes, as it represents 16-bit audio data.
JNIEXPORT jbyteArray JNICALL JNI_METHOD(processAudioBytes)(JNIEnv* env,
jobject instance,
jlong instance_ptr,
jbyteArray samples,
jint offset,
jint length);
// Tell the encoder that there will be no more samples.
JNIEXPORT jbyteArray JNICALL JNI_METHOD(flush)(JNIEnv* env, jobject instance,
jlong instance_ptr);
// Releases all resources.
JNIEXPORT void JNICALL JNI_METHOD(free)(JNIEnv* env, jobject instance,
jlong instance_ptr);
} // extern "C"
#endif // AUDIO_UTIL_JNI_OGG_OPUS_ENCODER_H_
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libogg/Makefile.am
================================================
## Process this file with automake to produce Makefile.in
oggincludedir = $(includedir)/ogg
ogginclude_HEADERS = ogg.h os_types.h
nodist_ogginclude_HEADERS = config_types.h
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libogg/config_types.h
================================================
#ifndef __CONFIG_TYPES_H__
#define __CONFIG_TYPES_H__
/* these are filled in by configure or cmake*/
#define INCLUDE_INTTYPES_H 1
#define INCLUDE_STDINT_H 1
#define INCLUDE_SYS_TYPES_H 1
#if INCLUDE_INTTYPES_H
# include
#endif
#if INCLUDE_STDINT_H
# include
#endif
#if INCLUDE_SYS_TYPES_H
# include
#endif
typedef int16_t ogg_int16_t;
typedef uint16_t ogg_uint16_t;
typedef int32_t ogg_int32_t;
typedef uint32_t ogg_uint32_t;
typedef int64_t ogg_int64_t;
typedef uint64_t ogg_uint64_t;
#endif
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libogg/config_types.h.in
================================================
#ifndef __CONFIG_TYPES_H__
#define __CONFIG_TYPES_H__
/* these are filled in by configure or cmake*/
#define INCLUDE_INTTYPES_H @INCLUDE_INTTYPES_H@
#define INCLUDE_STDINT_H @INCLUDE_STDINT_H@
#define INCLUDE_SYS_TYPES_H @INCLUDE_SYS_TYPES_H@
#if INCLUDE_INTTYPES_H
# include
#endif
#if INCLUDE_STDINT_H
# include
#endif
#if INCLUDE_SYS_TYPES_H
# include
#endif
typedef @SIZE16@ ogg_int16_t;
typedef @USIZE16@ ogg_uint16_t;
typedef @SIZE32@ ogg_int32_t;
typedef @USIZE32@ ogg_uint32_t;
typedef @SIZE64@ ogg_int64_t;
typedef @USIZE64@ ogg_uint64_t;
#endif
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libogg/ogg.h
================================================
/********************************************************************
* *
* THIS FILE IS PART OF THE OggVorbis SOFTWARE CODEC SOURCE CODE. *
* USE, DISTRIBUTION AND REPRODUCTION OF THIS LIBRARY SOURCE IS *
* GOVERNED BY A BSD-STYLE SOURCE LICENSE INCLUDED WITH THIS SOURCE *
* IN 'COPYING'. PLEASE READ THESE TERMS BEFORE DISTRIBUTING. *
* *
* THE OggVorbis SOURCE CODE IS (C) COPYRIGHT 1994-2007 *
* by the Xiph.Org Foundation http://www.xiph.org/ *
* *
********************************************************************
function: toplevel libogg include
********************************************************************/
#ifndef _OGG_H
#define _OGG_H
#ifdef __cplusplus
extern "C" {
#endif
#include
#include "os_types.h"
typedef struct {
void *iov_base;
size_t iov_len;
} ogg_iovec_t;
typedef struct {
long endbyte;
int endbit;
unsigned char *buffer;
unsigned char *ptr;
long storage;
} oggpack_buffer;
/* ogg_page is used to encapsulate the data in one Ogg bitstream page *****/
typedef struct {
unsigned char *header;
long header_len;
unsigned char *body;
long body_len;
} ogg_page;
/* ogg_stream_state contains the current encode/decode state of a logical
Ogg bitstream **********************************************************/
typedef struct {
unsigned char *body_data; /* bytes from packet bodies */
long body_storage; /* storage elements allocated */
long body_fill; /* elements stored; fill mark */
long body_returned; /* elements of fill returned */
int *lacing_vals; /* The values that will go to the segment table */
ogg_int64_t *granule_vals; /* granulepos values for headers. Not compact
this way, but it is simple coupled to the
lacing fifo */
long lacing_storage;
long lacing_fill;
long lacing_packet;
long lacing_returned;
unsigned char header[282]; /* working space for header encode */
int header_fill;
int e_o_s; /* set when we have buffered the last packet in the
logical bitstream */
int b_o_s; /* set after we've written the initial page
of a logical bitstream */
long serialno;
long pageno;
ogg_int64_t packetno; /* sequence number for decode; the framing
knows where there's a hole in the data,
but we need coupling so that the codec
(which is in a separate abstraction
layer) also knows about the gap */
ogg_int64_t granulepos;
} ogg_stream_state;
/* ogg_packet is used to encapsulate the data and metadata belonging
to a single raw Ogg/Vorbis packet *************************************/
typedef struct {
unsigned char *packet;
long bytes;
long b_o_s;
long e_o_s;
ogg_int64_t granulepos;
ogg_int64_t packetno; /* sequence number for decode; the framing
knows where there's a hole in the data,
but we need coupling so that the codec
(which is in a separate abstraction
layer) also knows about the gap */
} ogg_packet;
typedef struct {
unsigned char *data;
int storage;
int fill;
int returned;
int unsynced;
int headerbytes;
int bodybytes;
} ogg_sync_state;
/* Ogg BITSTREAM PRIMITIVES: bitstream ************************/
extern void oggpack_writeinit(oggpack_buffer *b);
extern int oggpack_writecheck(oggpack_buffer *b);
extern void oggpack_writetrunc(oggpack_buffer *b,long bits);
extern void oggpack_writealign(oggpack_buffer *b);
extern void oggpack_writecopy(oggpack_buffer *b,void *source,long bits);
extern void oggpack_reset(oggpack_buffer *b);
extern void oggpack_writeclear(oggpack_buffer *b);
extern void oggpack_readinit(oggpack_buffer *b,unsigned char *buf,int bytes);
extern void oggpack_write(oggpack_buffer *b,unsigned long value,int bits);
extern long oggpack_look(oggpack_buffer *b,int bits);
extern long oggpack_look1(oggpack_buffer *b);
extern void oggpack_adv(oggpack_buffer *b,int bits);
extern void oggpack_adv1(oggpack_buffer *b);
extern long oggpack_read(oggpack_buffer *b,int bits);
extern long oggpack_read1(oggpack_buffer *b);
extern long oggpack_bytes(oggpack_buffer *b);
extern long oggpack_bits(oggpack_buffer *b);
extern unsigned char *oggpack_get_buffer(oggpack_buffer *b);
extern void oggpackB_writeinit(oggpack_buffer *b);
extern int oggpackB_writecheck(oggpack_buffer *b);
extern void oggpackB_writetrunc(oggpack_buffer *b,long bits);
extern void oggpackB_writealign(oggpack_buffer *b);
extern void oggpackB_writecopy(oggpack_buffer *b,void *source,long bits);
extern void oggpackB_reset(oggpack_buffer *b);
extern void oggpackB_writeclear(oggpack_buffer *b);
extern void oggpackB_readinit(oggpack_buffer *b,unsigned char *buf,int bytes);
extern void oggpackB_write(oggpack_buffer *b,unsigned long value,int bits);
extern long oggpackB_look(oggpack_buffer *b,int bits);
extern long oggpackB_look1(oggpack_buffer *b);
extern void oggpackB_adv(oggpack_buffer *b,int bits);
extern void oggpackB_adv1(oggpack_buffer *b);
extern long oggpackB_read(oggpack_buffer *b,int bits);
extern long oggpackB_read1(oggpack_buffer *b);
extern long oggpackB_bytes(oggpack_buffer *b);
extern long oggpackB_bits(oggpack_buffer *b);
extern unsigned char *oggpackB_get_buffer(oggpack_buffer *b);
/* Ogg BITSTREAM PRIMITIVES: encoding **************************/
extern int ogg_stream_packetin(ogg_stream_state *os, ogg_packet *op);
extern int ogg_stream_iovecin(ogg_stream_state *os, ogg_iovec_t *iov,
int count, long e_o_s, ogg_int64_t granulepos);
extern int ogg_stream_pageout(ogg_stream_state *os, ogg_page *og);
extern int ogg_stream_pageout_fill(ogg_stream_state *os, ogg_page *og, int nfill);
extern int ogg_stream_flush(ogg_stream_state *os, ogg_page *og);
extern int ogg_stream_flush_fill(ogg_stream_state *os, ogg_page *og, int nfill);
/* Ogg BITSTREAM PRIMITIVES: decoding **************************/
extern int ogg_sync_init(ogg_sync_state *oy);
extern int ogg_sync_clear(ogg_sync_state *oy);
extern int ogg_sync_reset(ogg_sync_state *oy);
extern int ogg_sync_destroy(ogg_sync_state *oy);
extern int ogg_sync_check(ogg_sync_state *oy);
extern char *ogg_sync_buffer(ogg_sync_state *oy, long size);
extern int ogg_sync_wrote(ogg_sync_state *oy, long bytes);
extern long ogg_sync_pageseek(ogg_sync_state *oy,ogg_page *og);
extern int ogg_sync_pageout(ogg_sync_state *oy, ogg_page *og);
extern int ogg_stream_pagein(ogg_stream_state *os, ogg_page *og);
extern int ogg_stream_packetout(ogg_stream_state *os,ogg_packet *op);
extern int ogg_stream_packetpeek(ogg_stream_state *os,ogg_packet *op);
/* Ogg BITSTREAM PRIMITIVES: general ***************************/
extern int ogg_stream_init(ogg_stream_state *os,int serialno);
extern int ogg_stream_clear(ogg_stream_state *os);
extern int ogg_stream_reset(ogg_stream_state *os);
extern int ogg_stream_reset_serialno(ogg_stream_state *os,int serialno);
extern int ogg_stream_destroy(ogg_stream_state *os);
extern int ogg_stream_check(ogg_stream_state *os);
extern int ogg_stream_eos(ogg_stream_state *os);
extern void ogg_page_checksum_set(ogg_page *og);
extern int ogg_page_version(const ogg_page *og);
extern int ogg_page_continued(const ogg_page *og);
extern int ogg_page_bos(const ogg_page *og);
extern int ogg_page_eos(const ogg_page *og);
extern ogg_int64_t ogg_page_granulepos(const ogg_page *og);
extern int ogg_page_serialno(const ogg_page *og);
extern long ogg_page_pageno(const ogg_page *og);
extern int ogg_page_packets(const ogg_page *og);
extern void ogg_packet_clear(ogg_packet *op);
#ifdef __cplusplus
}
#endif
#endif /* _OGG_H */
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libogg/os_types.h
================================================
/********************************************************************
* *
* THIS FILE IS PART OF THE OggVorbis SOFTWARE CODEC SOURCE CODE. *
* USE, DISTRIBUTION AND REPRODUCTION OF THIS LIBRARY SOURCE IS *
* GOVERNED BY A BSD-STYLE SOURCE LICENSE INCLUDED WITH THIS SOURCE *
* IN 'COPYING'. PLEASE READ THESE TERMS BEFORE DISTRIBUTING. *
* *
* THE OggVorbis SOURCE CODE IS (C) COPYRIGHT 1994-2002 *
* by the Xiph.Org Foundation http://www.xiph.org/ *
* *
********************************************************************
function: Define a consistent set of types on each platform.
********************************************************************/
#ifndef _OS_TYPES_H
#define _OS_TYPES_H
/* make it easy on the folks that want to compile the libs with a
different malloc than stdlib */
#define _ogg_malloc malloc
#define _ogg_calloc calloc
#define _ogg_realloc realloc
#define _ogg_free free
#if defined(_WIN32)
# if defined(__CYGWIN__)
# include
typedef int16_t ogg_int16_t;
typedef uint16_t ogg_uint16_t;
typedef int32_t ogg_int32_t;
typedef uint32_t ogg_uint32_t;
typedef int64_t ogg_int64_t;
typedef uint64_t ogg_uint64_t;
# elif defined(__MINGW32__)
# include
typedef short ogg_int16_t;
typedef unsigned short ogg_uint16_t;
typedef int ogg_int32_t;
typedef unsigned int ogg_uint32_t;
typedef long long ogg_int64_t;
typedef unsigned long long ogg_uint64_t;
# elif defined(__MWERKS__)
typedef long long ogg_int64_t;
typedef unsigned long long ogg_uint64_t;
typedef int ogg_int32_t;
typedef unsigned int ogg_uint32_t;
typedef short ogg_int16_t;
typedef unsigned short ogg_uint16_t;
# else
# if defined(_MSC_VER) && (_MSC_VER >= 1800) /* MSVC 2013 and newer */
# include
typedef int16_t ogg_int16_t;
typedef uint16_t ogg_uint16_t;
typedef int32_t ogg_int32_t;
typedef uint32_t ogg_uint32_t;
typedef int64_t ogg_int64_t;
typedef uint64_t ogg_uint64_t;
# else
/* MSVC/Borland */
typedef __int64 ogg_int64_t;
typedef __int32 ogg_int32_t;
typedef unsigned __int32 ogg_uint32_t;
typedef unsigned __int64 ogg_uint64_t;
typedef __int16 ogg_int16_t;
typedef unsigned __int16 ogg_uint16_t;
# endif
# endif
#elif (defined(__APPLE__) && defined(__MACH__)) /* MacOS X Framework build */
# include
typedef int16_t ogg_int16_t;
typedef uint16_t ogg_uint16_t;
typedef int32_t ogg_int32_t;
typedef uint32_t ogg_uint32_t;
typedef int64_t ogg_int64_t;
typedef uint64_t ogg_uint64_t;
#elif defined(__HAIKU__)
/* Haiku */
# include
typedef short ogg_int16_t;
typedef unsigned short ogg_uint16_t;
typedef int ogg_int32_t;
typedef unsigned int ogg_uint32_t;
typedef long long ogg_int64_t;
typedef unsigned long long ogg_uint64_t;
#elif defined(__BEOS__)
/* Be */
# include
typedef int16_t ogg_int16_t;
typedef uint16_t ogg_uint16_t;
typedef int32_t ogg_int32_t;
typedef uint32_t ogg_uint32_t;
typedef int64_t ogg_int64_t;
typedef uint64_t ogg_uint64_t;
#elif defined (__EMX__)
/* OS/2 GCC */
typedef short ogg_int16_t;
typedef unsigned short ogg_uint16_t;
typedef int ogg_int32_t;
typedef unsigned int ogg_uint32_t;
typedef long long ogg_int64_t;
typedef unsigned long long ogg_uint64_t;
#elif defined (DJGPP)
/* DJGPP */
typedef short ogg_int16_t;
typedef int ogg_int32_t;
typedef unsigned int ogg_uint32_t;
typedef long long ogg_int64_t;
typedef unsigned long long ogg_uint64_t;
#elif defined(R5900)
/* PS2 EE */
typedef long ogg_int64_t;
typedef unsigned long ogg_uint64_t;
typedef int ogg_int32_t;
typedef unsigned ogg_uint32_t;
typedef short ogg_int16_t;
#elif defined(__SYMBIAN32__)
/* Symbian GCC */
typedef signed short ogg_int16_t;
typedef unsigned short ogg_uint16_t;
typedef signed int ogg_int32_t;
typedef unsigned int ogg_uint32_t;
typedef long long int ogg_int64_t;
typedef unsigned long long int ogg_uint64_t;
#elif defined(__TMS320C6X__)
/* TI C64x compiler */
typedef signed short ogg_int16_t;
typedef unsigned short ogg_uint16_t;
typedef signed int ogg_int32_t;
typedef unsigned int ogg_uint32_t;
typedef long long int ogg_int64_t;
typedef unsigned long long int ogg_uint64_t;
#else
# include "config_types.h"
#endif
#endif /* _OS_TYPES_H */
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libopus/opus.h
================================================
/* Copyright (c) 2010-2011 Xiph.Org Foundation, Skype Limited
Written by Jean-Marc Valin and Koen Vos */
/*
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/**
* @file opus.h
* @brief Opus reference implementation API
*/
#ifndef OPUS_H
#define OPUS_H
#include "opus_types.h"
#include "opus_defines.h"
#ifdef __cplusplus
extern "C" {
#endif
/**
* @mainpage Opus
*
* The Opus codec is designed for interactive speech and audio transmission over the Internet.
* It is designed by the IETF Codec Working Group and incorporates technology from
* Skype's SILK codec and Xiph.Org's CELT codec.
*
* The Opus codec is designed to handle a wide range of interactive audio applications,
* including Voice over IP, videoconferencing, in-game chat, and even remote live music
* performances. It can scale from low bit-rate narrowband speech to very high quality
* stereo music. Its main features are:
* @li Sampling rates from 8 to 48 kHz
* @li Bit-rates from 6 kb/s to 510 kb/s
* @li Support for both constant bit-rate (CBR) and variable bit-rate (VBR)
* @li Audio bandwidth from narrowband to full-band
* @li Support for speech and music
* @li Support for mono and stereo
* @li Support for multichannel (up to 255 channels)
* @li Frame sizes from 2.5 ms to 60 ms
* @li Good loss robustness and packet loss concealment (PLC)
* @li Floating point and fixed-point implementation
*
* Documentation sections:
* @li @ref opus_encoder
* @li @ref opus_decoder
* @li @ref opus_repacketizer
* @li @ref opus_multistream
* @li @ref opus_libinfo
* @li @ref opus_custom
*/
/** @defgroup opus_encoder Opus Encoder
* @{
*
* @brief This page describes the process and functions used to encode Opus.
*
* Since Opus is a stateful codec, the encoding process starts with creating an encoder
* state. This can be done with:
*
* @code
* int error;
* OpusEncoder *enc;
* enc = opus_encoder_create(Fs, channels, application, &error);
* @endcode
*
* From this point, @c enc can be used for encoding an audio stream. An encoder state
* @b must @b not be used for more than one stream at the same time. Similarly, the encoder
* state @b must @b not be re-initialized for each frame.
*
* While opus_encoder_create() allocates memory for the state, it's also possible
* to initialize pre-allocated memory:
*
* @code
* int size;
* int error;
* OpusEncoder *enc;
* size = opus_encoder_get_size(channels);
* enc = malloc(size);
* error = opus_encoder_init(enc, Fs, channels, application);
* @endcode
*
* where opus_encoder_get_size() returns the required size for the encoder state. Note that
* future versions of this code may change the size, so no assuptions should be made about it.
*
* The encoder state is always continuous in memory and only a shallow copy is sufficient
* to copy it (e.g. memcpy())
*
* It is possible to change some of the encoder's settings using the opus_encoder_ctl()
* interface. All these settings already default to the recommended value, so they should
* only be changed when necessary. The most common settings one may want to change are:
*
* @code
* opus_encoder_ctl(enc, OPUS_SET_BITRATE(bitrate));
* opus_encoder_ctl(enc, OPUS_SET_COMPLEXITY(complexity));
* opus_encoder_ctl(enc, OPUS_SET_SIGNAL(signal_type));
* @endcode
*
* where
*
* @arg bitrate is in bits per second (b/s)
* @arg complexity is a value from 1 to 10, where 1 is the lowest complexity and 10 is the highest
* @arg signal_type is either OPUS_AUTO (default), OPUS_SIGNAL_VOICE, or OPUS_SIGNAL_MUSIC
*
* See @ref opus_encoderctls and @ref opus_genericctls for a complete list of parameters that can be set or queried. Most parameters can be set or changed at any time during a stream.
*
* To encode a frame, opus_encode() or opus_encode_float() must be called with exactly one frame (2.5, 5, 10, 20, 40 or 60 ms) of audio data:
* @code
* len = opus_encode(enc, audio_frame, frame_size, packet, max_packet);
* @endcode
*
* where
*
*
audio_frame is the audio data in opus_int16 (or float for opus_encode_float())
*
frame_size is the duration of the frame in samples (per channel)
*
packet is the byte array to which the compressed data is written
*
max_packet is the maximum number of bytes that can be written in the packet (4000 bytes is recommended).
* Do not use max_packet to control VBR target bitrate, instead use the #OPUS_SET_BITRATE CTL.
*
*
* opus_encode() and opus_encode_float() return the number of bytes actually written to the packet.
* The return value can be negative, which indicates that an error has occurred. If the return value
* is 2 bytes or less, then the packet does not need to be transmitted (DTX).
*
* Once the encoder state if no longer needed, it can be destroyed with
*
* @code
* opus_encoder_destroy(enc);
* @endcode
*
* If the encoder was created with opus_encoder_init() rather than opus_encoder_create(),
* then no action is required aside from potentially freeing the memory that was manually
* allocated for it (calling free(enc) for the example above)
*
*/
/** Opus encoder state.
* This contains the complete state of an Opus encoder.
* It is position independent and can be freely copied.
* @see opus_encoder_create,opus_encoder_init
*/
typedef struct OpusEncoder OpusEncoder;
/** Gets the size of an OpusEncoder structure.
* @param[in] channels int: Number of channels.
* This must be 1 or 2.
* @returns The size in bytes.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_encoder_get_size(int channels);
/**
*/
/** Allocates and initializes an encoder state.
* There are three coding modes:
*
* @ref OPUS_APPLICATION_VOIP gives best quality at a given bitrate for voice
* signals. It enhances the input signal by high-pass filtering and
* emphasizing formants and harmonics. Optionally it includes in-band
* forward error correction to protect against packet loss. Use this
* mode for typical VoIP applications. Because of the enhancement,
* even at high bitrates the output may sound different from the input.
*
* @ref OPUS_APPLICATION_AUDIO gives best quality at a given bitrate for most
* non-voice signals like music. Use this mode for music and mixed
* (music/voice) content, broadcast, and applications requiring less
* than 15 ms of coding delay.
*
* @ref OPUS_APPLICATION_RESTRICTED_LOWDELAY configures low-delay mode that
* disables the speech-optimized mode in exchange for slightly reduced delay.
* This mode can only be set on an newly initialized or freshly reset encoder
* because it changes the codec delay.
*
* This is useful when the caller knows that the speech-optimized modes will not be needed (use with caution).
* @param [in] Fs opus_int32: Sampling rate of input signal (Hz)
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param [in] channels int: Number of channels (1 or 2) in input signal
* @param [in] application int: Coding mode (@ref OPUS_APPLICATION_VOIP/@ref OPUS_APPLICATION_AUDIO/@ref OPUS_APPLICATION_RESTRICTED_LOWDELAY)
* @param [out] error int*: @ref opus_errorcodes
* @note Regardless of the sampling rate and number channels selected, the Opus encoder
* can switch to a lower audio bandwidth or number of channels if the bitrate
* selected is too low. This also means that it is safe to always use 48 kHz stereo input
* and let the encoder optimize the encoding.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT OpusEncoder *opus_encoder_create(
opus_int32 Fs,
int channels,
int application,
int *error
);
/** Initializes a previously allocated encoder state
* The memory pointed to by st must be at least the size returned by opus_encoder_get_size().
* This is intended for applications which use their own allocator instead of malloc.
* @see opus_encoder_create(),opus_encoder_get_size()
* To reset a previously initialized state, use the #OPUS_RESET_STATE CTL.
* @param [in] st OpusEncoder*: Encoder state
* @param [in] Fs opus_int32: Sampling rate of input signal (Hz)
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param [in] channels int: Number of channels (1 or 2) in input signal
* @param [in] application int: Coding mode (OPUS_APPLICATION_VOIP/OPUS_APPLICATION_AUDIO/OPUS_APPLICATION_RESTRICTED_LOWDELAY)
* @retval #OPUS_OK Success or @ref opus_errorcodes
*/
OPUS_EXPORT int opus_encoder_init(
OpusEncoder *st,
opus_int32 Fs,
int channels,
int application
) OPUS_ARG_NONNULL(1);
/** Encodes an Opus frame.
* @param [in] st OpusEncoder*: Encoder state
* @param [in] pcm opus_int16*: Input signal (interleaved if 2 channels). length is frame_size*channels*sizeof(opus_int16)
* @param [in] frame_size int: Number of samples per channel in the
* input signal.
* This must be an Opus frame size for
* the encoder's sampling rate.
* For example, at 48 kHz the permitted
* values are 120, 240, 480, 960, 1920,
* and 2880.
* Passing in a duration of less than
* 10 ms (480 samples at 48 kHz) will
* prevent the encoder from using the LPC
* or hybrid modes.
* @param [out] data unsigned char*: Output payload.
* This must contain storage for at
* least \a max_data_bytes.
* @param [in] max_data_bytes opus_int32: Size of the allocated
* memory for the output
* payload. This may be
* used to impose an upper limit on
* the instant bitrate, but should
* not be used as the only bitrate
* control. Use #OPUS_SET_BITRATE to
* control the bitrate.
* @returns The length of the encoded packet (in bytes) on success or a
* negative error code (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_encode(
OpusEncoder *st,
const opus_int16 *pcm,
int frame_size,
unsigned char *data,
opus_int32 max_data_bytes
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2) OPUS_ARG_NONNULL(4);
/** Encodes an Opus frame from floating point input.
* @param [in] st OpusEncoder*: Encoder state
* @param [in] pcm float*: Input in float format (interleaved if 2 channels), with a normal range of +/-1.0.
* Samples with a range beyond +/-1.0 are supported but will
* be clipped by decoders using the integer API and should
* only be used if it is known that the far end supports
* extended dynamic range.
* length is frame_size*channels*sizeof(float)
* @param [in] frame_size int: Number of samples per channel in the
* input signal.
* This must be an Opus frame size for
* the encoder's sampling rate.
* For example, at 48 kHz the permitted
* values are 120, 240, 480, 960, 1920,
* and 2880.
* Passing in a duration of less than
* 10 ms (480 samples at 48 kHz) will
* prevent the encoder from using the LPC
* or hybrid modes.
* @param [out] data unsigned char*: Output payload.
* This must contain storage for at
* least \a max_data_bytes.
* @param [in] max_data_bytes opus_int32: Size of the allocated
* memory for the output
* payload. This may be
* used to impose an upper limit on
* the instant bitrate, but should
* not be used as the only bitrate
* control. Use #OPUS_SET_BITRATE to
* control the bitrate.
* @returns The length of the encoded packet (in bytes) on success or a
* negative error code (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_encode_float(
OpusEncoder *st,
const float *pcm,
int frame_size,
unsigned char *data,
opus_int32 max_data_bytes
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2) OPUS_ARG_NONNULL(4);
/** Frees an OpusEncoder allocated by opus_encoder_create().
* @param[in] st OpusEncoder*: State to be freed.
*/
OPUS_EXPORT void opus_encoder_destroy(OpusEncoder *st);
/** Perform a CTL function on an Opus encoder.
*
* Generally the request and subsequent arguments are generated
* by a convenience macro.
* @param st OpusEncoder*: Encoder state.
* @param request This and all remaining parameters should be replaced by one
* of the convenience macros in @ref opus_genericctls or
* @ref opus_encoderctls.
* @see opus_genericctls
* @see opus_encoderctls
*/
OPUS_EXPORT int opus_encoder_ctl(OpusEncoder *st, int request, ...) OPUS_ARG_NONNULL(1);
/**@}*/
/** @defgroup opus_decoder Opus Decoder
* @{
*
* @brief This page describes the process and functions used to decode Opus.
*
* The decoding process also starts with creating a decoder
* state. This can be done with:
* @code
* int error;
* OpusDecoder *dec;
* dec = opus_decoder_create(Fs, channels, &error);
* @endcode
* where
* @li Fs is the sampling rate and must be 8000, 12000, 16000, 24000, or 48000
* @li channels is the number of channels (1 or 2)
* @li error will hold the error code in case of failure (or #OPUS_OK on success)
* @li the return value is a newly created decoder state to be used for decoding
*
* While opus_decoder_create() allocates memory for the state, it's also possible
* to initialize pre-allocated memory:
* @code
* int size;
* int error;
* OpusDecoder *dec;
* size = opus_decoder_get_size(channels);
* dec = malloc(size);
* error = opus_decoder_init(dec, Fs, channels);
* @endcode
* where opus_decoder_get_size() returns the required size for the decoder state. Note that
* future versions of this code may change the size, so no assuptions should be made about it.
*
* The decoder state is always continuous in memory and only a shallow copy is sufficient
* to copy it (e.g. memcpy())
*
* To decode a frame, opus_decode() or opus_decode_float() must be called with a packet of compressed audio data:
* @code
* frame_size = opus_decode(dec, packet, len, decoded, max_size, 0);
* @endcode
* where
*
* @li packet is the byte array containing the compressed data
* @li len is the exact number of bytes contained in the packet
* @li decoded is the decoded audio data in opus_int16 (or float for opus_decode_float())
* @li max_size is the max duration of the frame in samples (per channel) that can fit into the decoded_frame array
*
* opus_decode() and opus_decode_float() return the number of samples (per channel) decoded from the packet.
* If that value is negative, then an error has occurred. This can occur if the packet is corrupted or if the audio
* buffer is too small to hold the decoded audio.
*
* Opus is a stateful codec with overlapping blocks and as a result Opus
* packets are not coded independently of each other. Packets must be
* passed into the decoder serially and in the correct order for a correct
* decode. Lost packets can be replaced with loss concealment by calling
* the decoder with a null pointer and zero length for the missing packet.
*
* A single codec state may only be accessed from a single thread at
* a time and any required locking must be performed by the caller. Separate
* streams must be decoded with separate decoder states and can be decoded
* in parallel unless the library was compiled with NONTHREADSAFE_PSEUDOSTACK
* defined.
*
*/
/** Opus decoder state.
* This contains the complete state of an Opus decoder.
* It is position independent and can be freely copied.
* @see opus_decoder_create,opus_decoder_init
*/
typedef struct OpusDecoder OpusDecoder;
/** Gets the size of an OpusDecoder structure.
* @param [in] channels int: Number of channels.
* This must be 1 or 2.
* @returns The size in bytes.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_decoder_get_size(int channels);
/** Allocates and initializes a decoder state.
* @param [in] Fs opus_int32: Sample rate to decode at (Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param [in] channels int: Number of channels (1 or 2) to decode
* @param [out] error int*: #OPUS_OK Success or @ref opus_errorcodes
*
* Internally Opus stores data at 48000 Hz, so that should be the default
* value for Fs. However, the decoder can efficiently decode to buffers
* at 8, 12, 16, and 24 kHz so if for some reason the caller cannot use
* data at the full sample rate, or knows the compressed data doesn't
* use the full frequency range, it can request decoding at a reduced
* rate. Likewise, the decoder is capable of filling in either mono or
* interleaved stereo pcm buffers, at the caller's request.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT OpusDecoder *opus_decoder_create(
opus_int32 Fs,
int channels,
int *error
);
/** Initializes a previously allocated decoder state.
* The state must be at least the size returned by opus_decoder_get_size().
* This is intended for applications which use their own allocator instead of malloc. @see opus_decoder_create,opus_decoder_get_size
* To reset a previously initialized state, use the #OPUS_RESET_STATE CTL.
* @param [in] st OpusDecoder*: Decoder state.
* @param [in] Fs opus_int32: Sampling rate to decode to (Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param [in] channels int: Number of channels (1 or 2) to decode
* @retval #OPUS_OK Success or @ref opus_errorcodes
*/
OPUS_EXPORT int opus_decoder_init(
OpusDecoder *st,
opus_int32 Fs,
int channels
) OPUS_ARG_NONNULL(1);
/** Decode an Opus packet.
* @param [in] st OpusDecoder*: Decoder state
* @param [in] data char*: Input payload. Use a NULL pointer to indicate packet loss
* @param [in] len opus_int32: Number of bytes in payload*
* @param [out] pcm opus_int16*: Output signal (interleaved if 2 channels). length
* is frame_size*channels*sizeof(opus_int16)
* @param [in] frame_size Number of samples per channel of available space in \a pcm.
* If this is less than the maximum packet duration (120ms; 5760 for 48kHz), this function will
* not be capable of decoding some packets. In the case of PLC (data==NULL) or FEC (decode_fec=1),
* then frame_size needs to be exactly the duration of audio that is missing, otherwise the
* decoder will not be in the optimal state to decode the next incoming packet. For the PLC and
* FEC cases, frame_size must be a multiple of 2.5 ms.
* @param [in] decode_fec int: Flag (0 or 1) to request that any in-band forward error correction data be
* decoded. If no such data is available, the frame is decoded as if it were lost.
* @returns Number of decoded samples or @ref opus_errorcodes
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_decode(
OpusDecoder *st,
const unsigned char *data,
opus_int32 len,
opus_int16 *pcm,
int frame_size,
int decode_fec
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Decode an Opus packet with floating point output.
* @param [in] st OpusDecoder*: Decoder state
* @param [in] data char*: Input payload. Use a NULL pointer to indicate packet loss
* @param [in] len opus_int32: Number of bytes in payload
* @param [out] pcm float*: Output signal (interleaved if 2 channels). length
* is frame_size*channels*sizeof(float)
* @param [in] frame_size Number of samples per channel of available space in \a pcm.
* If this is less than the maximum packet duration (120ms; 5760 for 48kHz), this function will
* not be capable of decoding some packets. In the case of PLC (data==NULL) or FEC (decode_fec=1),
* then frame_size needs to be exactly the duration of audio that is missing, otherwise the
* decoder will not be in the optimal state to decode the next incoming packet. For the PLC and
* FEC cases, frame_size must be a multiple of 2.5 ms.
* @param [in] decode_fec int: Flag (0 or 1) to request that any in-band forward error correction data be
* decoded. If no such data is available the frame is decoded as if it were lost.
* @returns Number of decoded samples or @ref opus_errorcodes
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_decode_float(
OpusDecoder *st,
const unsigned char *data,
opus_int32 len,
float *pcm,
int frame_size,
int decode_fec
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Perform a CTL function on an Opus decoder.
*
* Generally the request and subsequent arguments are generated
* by a convenience macro.
* @param st OpusDecoder*: Decoder state.
* @param request This and all remaining parameters should be replaced by one
* of the convenience macros in @ref opus_genericctls or
* @ref opus_decoderctls.
* @see opus_genericctls
* @see opus_decoderctls
*/
OPUS_EXPORT int opus_decoder_ctl(OpusDecoder *st, int request, ...) OPUS_ARG_NONNULL(1);
/** Frees an OpusDecoder allocated by opus_decoder_create().
* @param[in] st OpusDecoder*: State to be freed.
*/
OPUS_EXPORT void opus_decoder_destroy(OpusDecoder *st);
/** Parse an opus packet into one or more frames.
* Opus_decode will perform this operation internally so most applications do
* not need to use this function.
* This function does not copy the frames, the returned pointers are pointers into
* the input packet.
* @param [in] data char*: Opus packet to be parsed
* @param [in] len opus_int32: size of data
* @param [out] out_toc char*: TOC pointer
* @param [out] frames char*[48] encapsulated frames
* @param [out] size opus_int16[48] sizes of the encapsulated frames
* @param [out] payload_offset int*: returns the position of the payload within the packet (in bytes)
* @returns number of frames
*/
OPUS_EXPORT int opus_packet_parse(
const unsigned char *data,
opus_int32 len,
unsigned char *out_toc,
const unsigned char *frames[48],
opus_int16 size[48],
int *payload_offset
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(5);
/** Gets the bandwidth of an Opus packet.
* @param [in] data char*: Opus packet
* @retval OPUS_BANDWIDTH_NARROWBAND Narrowband (4kHz bandpass)
* @retval OPUS_BANDWIDTH_MEDIUMBAND Mediumband (6kHz bandpass)
* @retval OPUS_BANDWIDTH_WIDEBAND Wideband (8kHz bandpass)
* @retval OPUS_BANDWIDTH_SUPERWIDEBAND Superwideband (12kHz bandpass)
* @retval OPUS_BANDWIDTH_FULLBAND Fullband (20kHz bandpass)
* @retval OPUS_INVALID_PACKET The compressed data passed is corrupted or of an unsupported type
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_packet_get_bandwidth(const unsigned char *data) OPUS_ARG_NONNULL(1);
/** Gets the number of samples per frame from an Opus packet.
* @param [in] data char*: Opus packet.
* This must contain at least one byte of
* data.
* @param [in] Fs opus_int32: Sampling rate in Hz.
* This must be a multiple of 400, or
* inaccurate results will be returned.
* @returns Number of samples per frame.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_packet_get_samples_per_frame(const unsigned char *data, opus_int32 Fs) OPUS_ARG_NONNULL(1);
/** Gets the number of channels from an Opus packet.
* @param [in] data char*: Opus packet
* @returns Number of channels
* @retval OPUS_INVALID_PACKET The compressed data passed is corrupted or of an unsupported type
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_packet_get_nb_channels(const unsigned char *data) OPUS_ARG_NONNULL(1);
/** Gets the number of frames in an Opus packet.
* @param [in] packet char*: Opus packet
* @param [in] len opus_int32: Length of packet
* @returns Number of frames
* @retval OPUS_BAD_ARG Insufficient data was passed to the function
* @retval OPUS_INVALID_PACKET The compressed data passed is corrupted or of an unsupported type
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_packet_get_nb_frames(const unsigned char packet[], opus_int32 len) OPUS_ARG_NONNULL(1);
/** Gets the number of samples of an Opus packet.
* @param [in] packet char*: Opus packet
* @param [in] len opus_int32: Length of packet
* @param [in] Fs opus_int32: Sampling rate in Hz.
* This must be a multiple of 400, or
* inaccurate results will be returned.
* @returns Number of samples
* @retval OPUS_BAD_ARG Insufficient data was passed to the function
* @retval OPUS_INVALID_PACKET The compressed data passed is corrupted or of an unsupported type
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_packet_get_nb_samples(const unsigned char packet[], opus_int32 len, opus_int32 Fs) OPUS_ARG_NONNULL(1);
/** Gets the number of samples of an Opus packet.
* @param [in] dec OpusDecoder*: Decoder state
* @param [in] packet char*: Opus packet
* @param [in] len opus_int32: Length of packet
* @returns Number of samples
* @retval OPUS_BAD_ARG Insufficient data was passed to the function
* @retval OPUS_INVALID_PACKET The compressed data passed is corrupted or of an unsupported type
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_decoder_get_nb_samples(const OpusDecoder *dec, const unsigned char packet[], opus_int32 len) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2);
/** Applies soft-clipping to bring a float signal within the [-1,1] range. If
* the signal is already in that range, nothing is done. If there are values
* outside of [-1,1], then the signal is clipped as smoothly as possible to
* both fit in the range and avoid creating excessive distortion in the
* process.
* @param [in,out] pcm float*: Input PCM and modified PCM
* @param [in] frame_size int Number of samples per channel to process
* @param [in] channels int: Number of channels
* @param [in,out] softclip_mem float*: State memory for the soft clipping process (one float per channel, initialized to zero)
*/
OPUS_EXPORT void opus_pcm_soft_clip(float *pcm, int frame_size, int channels, float *softclip_mem);
/**@}*/
/** @defgroup opus_repacketizer Repacketizer
* @{
*
* The repacketizer can be used to merge multiple Opus packets into a single
* packet or alternatively to split Opus packets that have previously been
* merged. Splitting valid Opus packets is always guaranteed to succeed,
* whereas merging valid packets only succeeds if all frames have the same
* mode, bandwidth, and frame size, and when the total duration of the merged
* packet is no more than 120 ms. The 120 ms limit comes from the
* specification and limits decoder memory requirements at a point where
* framing overhead becomes negligible.
*
* The repacketizer currently only operates on elementary Opus
* streams. It will not manipualte multistream packets successfully, except in
* the degenerate case where they consist of data from a single stream.
*
* The repacketizing process starts with creating a repacketizer state, either
* by calling opus_repacketizer_create() or by allocating the memory yourself,
* e.g.,
* @code
* OpusRepacketizer *rp;
* rp = (OpusRepacketizer*)malloc(opus_repacketizer_get_size());
* if (rp != NULL)
* opus_repacketizer_init(rp);
* @endcode
*
* Then the application should submit packets with opus_repacketizer_cat(),
* extract new packets with opus_repacketizer_out() or
* opus_repacketizer_out_range(), and then reset the state for the next set of
* input packets via opus_repacketizer_init().
*
* For example, to split a sequence of packets into individual frames:
* @code
* unsigned char *data;
* int len;
* while (get_next_packet(&data, &len))
* {
* unsigned char out[1276];
* opus_int32 out_len;
* int nb_frames;
* int err;
* int i;
* err = opus_repacketizer_cat(rp, data, len);
* if (err != OPUS_OK)
* {
* release_packet(data);
* return err;
* }
* nb_frames = opus_repacketizer_get_nb_frames(rp);
* for (i = 0; i < nb_frames; i++)
* {
* out_len = opus_repacketizer_out_range(rp, i, i+1, out, sizeof(out));
* if (out_len < 0)
* {
* release_packet(data);
* return (int)out_len;
* }
* output_next_packet(out, out_len);
* }
* opus_repacketizer_init(rp);
* release_packet(data);
* }
* @endcode
*
* Alternatively, to combine a sequence of frames into packets that each
* contain up to TARGET_DURATION_MS milliseconds of data:
* @code
* // The maximum number of packets with duration TARGET_DURATION_MS occurs
* // when the frame size is 2.5 ms, for a total of (TARGET_DURATION_MS*2/5)
* // packets.
* unsigned char *data[(TARGET_DURATION_MS*2/5)+1];
* opus_int32 len[(TARGET_DURATION_MS*2/5)+1];
* int nb_packets;
* unsigned char out[1277*(TARGET_DURATION_MS*2/2)];
* opus_int32 out_len;
* int prev_toc;
* nb_packets = 0;
* while (get_next_packet(data+nb_packets, len+nb_packets))
* {
* int nb_frames;
* int err;
* nb_frames = opus_packet_get_nb_frames(data[nb_packets], len[nb_packets]);
* if (nb_frames < 1)
* {
* release_packets(data, nb_packets+1);
* return nb_frames;
* }
* nb_frames += opus_repacketizer_get_nb_frames(rp);
* // If adding the next packet would exceed our target, or it has an
* // incompatible TOC sequence, output the packets we already have before
* // submitting it.
* // N.B., The nb_packets > 0 check ensures we've submitted at least one
* // packet since the last call to opus_repacketizer_init(). Otherwise a
* // single packet longer than TARGET_DURATION_MS would cause us to try to
* // output an (invalid) empty packet. It also ensures that prev_toc has
* // been set to a valid value. Additionally, len[nb_packets] > 0 is
* // guaranteed by the call to opus_packet_get_nb_frames() above, so the
* // reference to data[nb_packets][0] should be valid.
* if (nb_packets > 0 && (
* ((prev_toc & 0xFC) != (data[nb_packets][0] & 0xFC)) ||
* opus_packet_get_samples_per_frame(data[nb_packets], 48000)*nb_frames >
* TARGET_DURATION_MS*48))
* {
* out_len = opus_repacketizer_out(rp, out, sizeof(out));
* if (out_len < 0)
* {
* release_packets(data, nb_packets+1);
* return (int)out_len;
* }
* output_next_packet(out, out_len);
* opus_repacketizer_init(rp);
* release_packets(data, nb_packets);
* data[0] = data[nb_packets];
* len[0] = len[nb_packets];
* nb_packets = 0;
* }
* err = opus_repacketizer_cat(rp, data[nb_packets], len[nb_packets]);
* if (err != OPUS_OK)
* {
* release_packets(data, nb_packets+1);
* return err;
* }
* prev_toc = data[nb_packets][0];
* nb_packets++;
* }
* // Output the final, partial packet.
* if (nb_packets > 0)
* {
* out_len = opus_repacketizer_out(rp, out, sizeof(out));
* release_packets(data, nb_packets);
* if (out_len < 0)
* return (int)out_len;
* output_next_packet(out, out_len);
* }
* @endcode
*
* An alternate way of merging packets is to simply call opus_repacketizer_cat()
* unconditionally until it fails. At that point, the merged packet can be
* obtained with opus_repacketizer_out() and the input packet for which
* opus_repacketizer_cat() needs to be re-added to a newly reinitialized
* repacketizer state.
*/
typedef struct OpusRepacketizer OpusRepacketizer;
/** Gets the size of an OpusRepacketizer structure.
* @returns The size in bytes.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_repacketizer_get_size(void);
/** (Re)initializes a previously allocated repacketizer state.
* The state must be at least the size returned by opus_repacketizer_get_size().
* This can be used for applications which use their own allocator instead of
* malloc().
* It must also be called to reset the queue of packets waiting to be
* repacketized, which is necessary if the maximum packet duration of 120 ms
* is reached or if you wish to submit packets with a different Opus
* configuration (coding mode, audio bandwidth, frame size, or channel count).
* Failure to do so will prevent a new packet from being added with
* opus_repacketizer_cat().
* @see opus_repacketizer_create
* @see opus_repacketizer_get_size
* @see opus_repacketizer_cat
* @param rp OpusRepacketizer*: The repacketizer state to
* (re)initialize.
* @returns A pointer to the same repacketizer state that was passed in.
*/
OPUS_EXPORT OpusRepacketizer *opus_repacketizer_init(OpusRepacketizer *rp) OPUS_ARG_NONNULL(1);
/** Allocates memory and initializes the new repacketizer with
* opus_repacketizer_init().
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT OpusRepacketizer *opus_repacketizer_create(void);
/** Frees an OpusRepacketizer allocated by
* opus_repacketizer_create().
* @param[in] rp OpusRepacketizer*: State to be freed.
*/
OPUS_EXPORT void opus_repacketizer_destroy(OpusRepacketizer *rp);
/** Add a packet to the current repacketizer state.
* This packet must match the configuration of any packets already submitted
* for repacketization since the last call to opus_repacketizer_init().
* This means that it must have the same coding mode, audio bandwidth, frame
* size, and channel count.
* This can be checked in advance by examining the top 6 bits of the first
* byte of the packet, and ensuring they match the top 6 bits of the first
* byte of any previously submitted packet.
* The total duration of audio in the repacketizer state also must not exceed
* 120 ms, the maximum duration of a single packet, after adding this packet.
*
* The contents of the current repacketizer state can be extracted into new
* packets using opus_repacketizer_out() or opus_repacketizer_out_range().
*
* In order to add a packet with a different configuration or to add more
* audio beyond 120 ms, you must clear the repacketizer state by calling
* opus_repacketizer_init().
* If a packet is too large to add to the current repacketizer state, no part
* of it is added, even if it contains multiple frames, some of which might
* fit.
* If you wish to be able to add parts of such packets, you should first use
* another repacketizer to split the packet into pieces and add them
* individually.
* @see opus_repacketizer_out_range
* @see opus_repacketizer_out
* @see opus_repacketizer_init
* @param rp OpusRepacketizer*: The repacketizer state to which to
* add the packet.
* @param[in] data const unsigned char*: The packet data.
* The application must ensure
* this pointer remains valid
* until the next call to
* opus_repacketizer_init() or
* opus_repacketizer_destroy().
* @param len opus_int32: The number of bytes in the packet data.
* @returns An error code indicating whether or not the operation succeeded.
* @retval #OPUS_OK The packet's contents have been added to the repacketizer
* state.
* @retval #OPUS_INVALID_PACKET The packet did not have a valid TOC sequence,
* the packet's TOC sequence was not compatible
* with previously submitted packets (because
* the coding mode, audio bandwidth, frame size,
* or channel count did not match), or adding
* this packet would increase the total amount of
* audio stored in the repacketizer state to more
* than 120 ms.
*/
OPUS_EXPORT int opus_repacketizer_cat(OpusRepacketizer *rp, const unsigned char *data, opus_int32 len) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2);
/** Construct a new packet from data previously submitted to the repacketizer
* state via opus_repacketizer_cat().
* @param rp OpusRepacketizer*: The repacketizer state from which to
* construct the new packet.
* @param begin int: The index of the first frame in the current
* repacketizer state to include in the output.
* @param end int: One past the index of the last frame in the
* current repacketizer state to include in the
* output.
* @param[out] data const unsigned char*: The buffer in which to
* store the output packet.
* @param maxlen opus_int32: The maximum number of bytes to store in
* the output buffer. In order to guarantee
* success, this should be at least
* 1276 for a single frame,
* or for multiple frames,
* 1277*(end-begin).
* However, 1*(end-begin) plus
* the size of all packet data submitted to
* the repacketizer since the last call to
* opus_repacketizer_init() or
* opus_repacketizer_create() is also
* sufficient, and possibly much smaller.
* @returns The total size of the output packet on success, or an error code
* on failure.
* @retval #OPUS_BAD_ARG [begin,end) was an invalid range of
* frames (begin < 0, begin >= end, or end >
* opus_repacketizer_get_nb_frames()).
* @retval #OPUS_BUFFER_TOO_SMALL \a maxlen was insufficient to contain the
* complete output packet.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_repacketizer_out_range(OpusRepacketizer *rp, int begin, int end, unsigned char *data, opus_int32 maxlen) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Return the total number of frames contained in packet data submitted to
* the repacketizer state so far via opus_repacketizer_cat() since the last
* call to opus_repacketizer_init() or opus_repacketizer_create().
* This defines the valid range of packets that can be extracted with
* opus_repacketizer_out_range() or opus_repacketizer_out().
* @param rp OpusRepacketizer*: The repacketizer state containing the
* frames.
* @returns The total number of frames contained in the packet data submitted
* to the repacketizer state.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_repacketizer_get_nb_frames(OpusRepacketizer *rp) OPUS_ARG_NONNULL(1);
/** Construct a new packet from data previously submitted to the repacketizer
* state via opus_repacketizer_cat().
* This is a convenience routine that returns all the data submitted so far
* in a single packet.
* It is equivalent to calling
* @code
* opus_repacketizer_out_range(rp, 0, opus_repacketizer_get_nb_frames(rp),
* data, maxlen)
* @endcode
* @param rp OpusRepacketizer*: The repacketizer state from which to
* construct the new packet.
* @param[out] data const unsigned char*: The buffer in which to
* store the output packet.
* @param maxlen opus_int32: The maximum number of bytes to store in
* the output buffer. In order to guarantee
* success, this should be at least
* 1277*opus_repacketizer_get_nb_frames(rp).
* However,
* 1*opus_repacketizer_get_nb_frames(rp)
* plus the size of all packet data
* submitted to the repacketizer since the
* last call to opus_repacketizer_init() or
* opus_repacketizer_create() is also
* sufficient, and possibly much smaller.
* @returns The total size of the output packet on success, or an error code
* on failure.
* @retval #OPUS_BUFFER_TOO_SMALL \a maxlen was insufficient to contain the
* complete output packet.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_repacketizer_out(OpusRepacketizer *rp, unsigned char *data, opus_int32 maxlen) OPUS_ARG_NONNULL(1);
/** Pads a given Opus packet to a larger size (possibly changing the TOC sequence).
* @param[in,out] data const unsigned char*: The buffer containing the
* packet to pad.
* @param len opus_int32: The size of the packet.
* This must be at least 1.
* @param new_len opus_int32: The desired size of the packet after padding.
* This must be at least as large as len.
* @returns an error code
* @retval #OPUS_OK \a on success.
* @retval #OPUS_BAD_ARG \a len was less than 1 or new_len was less than len.
* @retval #OPUS_INVALID_PACKET \a data did not contain a valid Opus packet.
*/
OPUS_EXPORT int opus_packet_pad(unsigned char *data, opus_int32 len, opus_int32 new_len);
/** Remove all padding from a given Opus packet and rewrite the TOC sequence to
* minimize space usage.
* @param[in,out] data const unsigned char*: The buffer containing the
* packet to strip.
* @param len opus_int32: The size of the packet.
* This must be at least 1.
* @returns The new size of the output packet on success, or an error code
* on failure.
* @retval #OPUS_BAD_ARG \a len was less than 1.
* @retval #OPUS_INVALID_PACKET \a data did not contain a valid Opus packet.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_packet_unpad(unsigned char *data, opus_int32 len);
/** Pads a given Opus multi-stream packet to a larger size (possibly changing the TOC sequence).
* @param[in,out] data const unsigned char*: The buffer containing the
* packet to pad.
* @param len opus_int32: The size of the packet.
* This must be at least 1.
* @param new_len opus_int32: The desired size of the packet after padding.
* This must be at least 1.
* @param nb_streams opus_int32: The number of streams (not channels) in the packet.
* This must be at least as large as len.
* @returns an error code
* @retval #OPUS_OK \a on success.
* @retval #OPUS_BAD_ARG \a len was less than 1.
* @retval #OPUS_INVALID_PACKET \a data did not contain a valid Opus packet.
*/
OPUS_EXPORT int opus_multistream_packet_pad(unsigned char *data, opus_int32 len, opus_int32 new_len, int nb_streams);
/** Remove all padding from a given Opus multi-stream packet and rewrite the TOC sequence to
* minimize space usage.
* @param[in,out] data const unsigned char*: The buffer containing the
* packet to strip.
* @param len opus_int32: The size of the packet.
* This must be at least 1.
* @param nb_streams opus_int32: The number of streams (not channels) in the packet.
* This must be at least 1.
* @returns The new size of the output packet on success, or an error code
* on failure.
* @retval #OPUS_BAD_ARG \a len was less than 1 or new_len was less than len.
* @retval #OPUS_INVALID_PACKET \a data did not contain a valid Opus packet.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_multistream_packet_unpad(unsigned char *data, opus_int32 len, int nb_streams);
/**@}*/
#ifdef __cplusplus
}
#endif
#endif /* OPUS_H */
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libopus/opus_custom.h
================================================
/* Copyright (c) 2007-2008 CSIRO
Copyright (c) 2007-2009 Xiph.Org Foundation
Copyright (c) 2008-2012 Gregory Maxwell
Written by Jean-Marc Valin and Gregory Maxwell */
/*
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/**
@file opus_custom.h
@brief Opus-Custom reference implementation API
*/
#ifndef OPUS_CUSTOM_H
#define OPUS_CUSTOM_H
#include "opus_defines.h"
#ifdef __cplusplus
extern "C" {
#endif
#ifdef CUSTOM_MODES
# define OPUS_CUSTOM_EXPORT OPUS_EXPORT
# define OPUS_CUSTOM_EXPORT_STATIC OPUS_EXPORT
#else
# define OPUS_CUSTOM_EXPORT
# ifdef OPUS_BUILD
# define OPUS_CUSTOM_EXPORT_STATIC static OPUS_INLINE
# else
# define OPUS_CUSTOM_EXPORT_STATIC
# endif
#endif
/** @defgroup opus_custom Opus Custom
* @{
* Opus Custom is an optional part of the Opus specification and
* reference implementation which uses a distinct API from the regular
* API and supports frame sizes that are not normally supported.\ Use
* of Opus Custom is discouraged for all but very special applications
* for which a frame size different from 2.5, 5, 10, or 20 ms is needed
* (for either complexity or latency reasons) and where interoperability
* is less important.
*
* In addition to the interoperability limitations the use of Opus custom
* disables a substantial chunk of the codec and generally lowers the
* quality available at a given bitrate. Normally when an application needs
* a different frame size from the codec it should buffer to match the
* sizes but this adds a small amount of delay which may be important
* in some very low latency applications. Some transports (especially
* constant rate RF transports) may also work best with frames of
* particular durations.
*
* Libopus only supports custom modes if they are enabled at compile time.
*
* The Opus Custom API is similar to the regular API but the
* @ref opus_encoder_create and @ref opus_decoder_create calls take
* an additional mode parameter which is a structure produced by
* a call to @ref opus_custom_mode_create. Both the encoder and decoder
* must create a mode using the same sample rate (fs) and frame size
* (frame size) so these parameters must either be signaled out of band
* or fixed in a particular implementation.
*
* Similar to regular Opus the custom modes support on the fly frame size
* switching, but the sizes available depend on the particular frame size in
* use. For some initial frame sizes on a single on the fly size is available.
*/
/** Contains the state of an encoder. One encoder state is needed
for each stream. It is initialized once at the beginning of the
stream. Do *not* re-initialize the state for every frame.
@brief Encoder state
*/
typedef struct OpusCustomEncoder OpusCustomEncoder;
/** State of the decoder. One decoder state is needed for each stream.
It is initialized once at the beginning of the stream. Do *not*
re-initialize the state for every frame.
@brief Decoder state
*/
typedef struct OpusCustomDecoder OpusCustomDecoder;
/** The mode contains all the information necessary to create an
encoder. Both the encoder and decoder need to be initialized
with exactly the same mode, otherwise the output will be
corrupted.
@brief Mode configuration
*/
typedef struct OpusCustomMode OpusCustomMode;
/** Creates a new mode struct. This will be passed to an encoder or
* decoder. The mode MUST NOT BE DESTROYED until the encoders and
* decoders that use it are destroyed as well.
* @param [in] Fs int: Sampling rate (8000 to 96000 Hz)
* @param [in] frame_size int: Number of samples (per channel) to encode in each
* packet (64 - 1024, prime factorization must contain zero or more 2s, 3s, or 5s and no other primes)
* @param [out] error int*: Returned error code (if NULL, no error will be returned)
* @return A newly created mode
*/
OPUS_CUSTOM_EXPORT OPUS_WARN_UNUSED_RESULT OpusCustomMode *opus_custom_mode_create(opus_int32 Fs, int frame_size, int *error);
/** Destroys a mode struct. Only call this after all encoders and
* decoders using this mode are destroyed as well.
* @param [in] mode OpusCustomMode*: Mode to be freed.
*/
OPUS_CUSTOM_EXPORT void opus_custom_mode_destroy(OpusCustomMode *mode);
#if !defined(OPUS_BUILD) || defined(CELT_ENCODER_C)
/* Encoder */
/** Gets the size of an OpusCustomEncoder structure.
* @param [in] mode OpusCustomMode *: Mode configuration
* @param [in] channels int: Number of channels
* @returns size
*/
OPUS_CUSTOM_EXPORT_STATIC OPUS_WARN_UNUSED_RESULT int opus_custom_encoder_get_size(
const OpusCustomMode *mode,
int channels
) OPUS_ARG_NONNULL(1);
# ifdef CUSTOM_MODES
/** Initializes a previously allocated encoder state
* The memory pointed to by st must be the size returned by opus_custom_encoder_get_size.
* This is intended for applications which use their own allocator instead of malloc.
* @see opus_custom_encoder_create(),opus_custom_encoder_get_size()
* To reset a previously initialized state use the OPUS_RESET_STATE CTL.
* @param [in] st OpusCustomEncoder*: Encoder state
* @param [in] mode OpusCustomMode *: Contains all the information about the characteristics of
* the stream (must be the same characteristics as used for the
* decoder)
* @param [in] channels int: Number of channels
* @return OPUS_OK Success or @ref opus_errorcodes
*/
OPUS_CUSTOM_EXPORT int opus_custom_encoder_init(
OpusCustomEncoder *st,
const OpusCustomMode *mode,
int channels
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2);
# endif
#endif
/** Creates a new encoder state. Each stream needs its own encoder
* state (can't be shared across simultaneous streams).
* @param [in] mode OpusCustomMode*: Contains all the information about the characteristics of
* the stream (must be the same characteristics as used for the
* decoder)
* @param [in] channels int: Number of channels
* @param [out] error int*: Returns an error code
* @return Newly created encoder state.
*/
OPUS_CUSTOM_EXPORT OPUS_WARN_UNUSED_RESULT OpusCustomEncoder *opus_custom_encoder_create(
const OpusCustomMode *mode,
int channels,
int *error
) OPUS_ARG_NONNULL(1);
/** Destroys a an encoder state.
* @param[in] st OpusCustomEncoder*: State to be freed.
*/
OPUS_CUSTOM_EXPORT void opus_custom_encoder_destroy(OpusCustomEncoder *st);
/** Encodes a frame of audio.
* @param [in] st OpusCustomEncoder*: Encoder state
* @param [in] pcm float*: PCM audio in float format, with a normal range of +/-1.0.
* Samples with a range beyond +/-1.0 are supported but will
* be clipped by decoders using the integer API and should
* only be used if it is known that the far end supports
* extended dynamic range. There must be exactly
* frame_size samples per channel.
* @param [in] frame_size int: Number of samples per frame of input signal
* @param [out] compressed char *: The compressed data is written here. This may not alias pcm and must be at least maxCompressedBytes long.
* @param [in] maxCompressedBytes int: Maximum number of bytes to use for compressing the frame
* (can change from one frame to another)
* @return Number of bytes written to "compressed".
* If negative, an error has occurred (see error codes). It is IMPORTANT that
* the length returned be somehow transmitted to the decoder. Otherwise, no
* decoding is possible.
*/
OPUS_CUSTOM_EXPORT OPUS_WARN_UNUSED_RESULT int opus_custom_encode_float(
OpusCustomEncoder *st,
const float *pcm,
int frame_size,
unsigned char *compressed,
int maxCompressedBytes
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2) OPUS_ARG_NONNULL(4);
/** Encodes a frame of audio.
* @param [in] st OpusCustomEncoder*: Encoder state
* @param [in] pcm opus_int16*: PCM audio in signed 16-bit format (native endian).
* There must be exactly frame_size samples per channel.
* @param [in] frame_size int: Number of samples per frame of input signal
* @param [out] compressed char *: The compressed data is written here. This may not alias pcm and must be at least maxCompressedBytes long.
* @param [in] maxCompressedBytes int: Maximum number of bytes to use for compressing the frame
* (can change from one frame to another)
* @return Number of bytes written to "compressed".
* If negative, an error has occurred (see error codes). It is IMPORTANT that
* the length returned be somehow transmitted to the decoder. Otherwise, no
* decoding is possible.
*/
OPUS_CUSTOM_EXPORT OPUS_WARN_UNUSED_RESULT int opus_custom_encode(
OpusCustomEncoder *st,
const opus_int16 *pcm,
int frame_size,
unsigned char *compressed,
int maxCompressedBytes
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2) OPUS_ARG_NONNULL(4);
/** Perform a CTL function on an Opus custom encoder.
*
* Generally the request and subsequent arguments are generated
* by a convenience macro.
* @see opus_encoderctls
*/
OPUS_CUSTOM_EXPORT int opus_custom_encoder_ctl(OpusCustomEncoder * OPUS_RESTRICT st, int request, ...) OPUS_ARG_NONNULL(1);
#if !defined(OPUS_BUILD) || defined(CELT_DECODER_C)
/* Decoder */
/** Gets the size of an OpusCustomDecoder structure.
* @param [in] mode OpusCustomMode *: Mode configuration
* @param [in] channels int: Number of channels
* @returns size
*/
OPUS_CUSTOM_EXPORT_STATIC OPUS_WARN_UNUSED_RESULT int opus_custom_decoder_get_size(
const OpusCustomMode *mode,
int channels
) OPUS_ARG_NONNULL(1);
/** Initializes a previously allocated decoder state
* The memory pointed to by st must be the size returned by opus_custom_decoder_get_size.
* This is intended for applications which use their own allocator instead of malloc.
* @see opus_custom_decoder_create(),opus_custom_decoder_get_size()
* To reset a previously initialized state use the OPUS_RESET_STATE CTL.
* @param [in] st OpusCustomDecoder*: Decoder state
* @param [in] mode OpusCustomMode *: Contains all the information about the characteristics of
* the stream (must be the same characteristics as used for the
* encoder)
* @param [in] channels int: Number of channels
* @return OPUS_OK Success or @ref opus_errorcodes
*/
OPUS_CUSTOM_EXPORT_STATIC int opus_custom_decoder_init(
OpusCustomDecoder *st,
const OpusCustomMode *mode,
int channels
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2);
#endif
/** Creates a new decoder state. Each stream needs its own decoder state (can't
* be shared across simultaneous streams).
* @param [in] mode OpusCustomMode: Contains all the information about the characteristics of the
* stream (must be the same characteristics as used for the encoder)
* @param [in] channels int: Number of channels
* @param [out] error int*: Returns an error code
* @return Newly created decoder state.
*/
OPUS_CUSTOM_EXPORT OPUS_WARN_UNUSED_RESULT OpusCustomDecoder *opus_custom_decoder_create(
const OpusCustomMode *mode,
int channels,
int *error
) OPUS_ARG_NONNULL(1);
/** Destroys a an decoder state.
* @param[in] st OpusCustomDecoder*: State to be freed.
*/
OPUS_CUSTOM_EXPORT void opus_custom_decoder_destroy(OpusCustomDecoder *st);
/** Decode an opus custom frame with floating point output
* @param [in] st OpusCustomDecoder*: Decoder state
* @param [in] data char*: Input payload. Use a NULL pointer to indicate packet loss
* @param [in] len int: Number of bytes in payload
* @param [out] pcm float*: Output signal (interleaved if 2 channels). length
* is frame_size*channels*sizeof(float)
* @param [in] frame_size Number of samples per channel of available space in *pcm.
* @returns Number of decoded samples or @ref opus_errorcodes
*/
OPUS_CUSTOM_EXPORT OPUS_WARN_UNUSED_RESULT int opus_custom_decode_float(
OpusCustomDecoder *st,
const unsigned char *data,
int len,
float *pcm,
int frame_size
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Decode an opus custom frame
* @param [in] st OpusCustomDecoder*: Decoder state
* @param [in] data char*: Input payload. Use a NULL pointer to indicate packet loss
* @param [in] len int: Number of bytes in payload
* @param [out] pcm opus_int16*: Output signal (interleaved if 2 channels). length
* is frame_size*channels*sizeof(opus_int16)
* @param [in] frame_size Number of samples per channel of available space in *pcm.
* @returns Number of decoded samples or @ref opus_errorcodes
*/
OPUS_CUSTOM_EXPORT OPUS_WARN_UNUSED_RESULT int opus_custom_decode(
OpusCustomDecoder *st,
const unsigned char *data,
int len,
opus_int16 *pcm,
int frame_size
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Perform a CTL function on an Opus custom decoder.
*
* Generally the request and subsequent arguments are generated
* by a convenience macro.
* @see opus_genericctls
*/
OPUS_CUSTOM_EXPORT int opus_custom_decoder_ctl(OpusCustomDecoder * OPUS_RESTRICT st, int request, ...) OPUS_ARG_NONNULL(1);
/**@}*/
#ifdef __cplusplus
}
#endif
#endif /* OPUS_CUSTOM_H */
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libopus/opus_defines.h
================================================
/* Copyright (c) 2010-2011 Xiph.Org Foundation, Skype Limited
Written by Jean-Marc Valin and Koen Vos */
/*
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/**
* @file opus_defines.h
* @brief Opus reference implementation constants
*/
#ifndef OPUS_DEFINES_H
#define OPUS_DEFINES_H
#include "opus_types.h"
#ifdef __cplusplus
extern "C" {
#endif
/** @defgroup opus_errorcodes Error codes
* @{
*/
/** No error @hideinitializer*/
#define OPUS_OK 0
/** One or more invalid/out of range arguments @hideinitializer*/
#define OPUS_BAD_ARG -1
/** Not enough bytes allocated in the buffer @hideinitializer*/
#define OPUS_BUFFER_TOO_SMALL -2
/** An internal error was detected @hideinitializer*/
#define OPUS_INTERNAL_ERROR -3
/** The compressed data passed is corrupted @hideinitializer*/
#define OPUS_INVALID_PACKET -4
/** Invalid/unsupported request number @hideinitializer*/
#define OPUS_UNIMPLEMENTED -5
/** An encoder or decoder structure is invalid or already freed @hideinitializer*/
#define OPUS_INVALID_STATE -6
/** Memory allocation has failed @hideinitializer*/
#define OPUS_ALLOC_FAIL -7
/**@}*/
/** @cond OPUS_INTERNAL_DOC */
/**Export control for opus functions */
#ifndef OPUS_EXPORT
# if defined(WIN32)
# if defined(OPUS_BUILD) && defined(DLL_EXPORT)
# define OPUS_EXPORT __declspec(dllexport)
# else
# define OPUS_EXPORT
# endif
# elif defined(__GNUC__) && defined(OPUS_BUILD)
# define OPUS_EXPORT __attribute__ ((visibility ("default")))
# else
# define OPUS_EXPORT
# endif
#endif
# if !defined(OPUS_GNUC_PREREQ)
# if defined(__GNUC__)&&defined(__GNUC_MINOR__)
# define OPUS_GNUC_PREREQ(_maj,_min) \
((__GNUC__<<16)+__GNUC_MINOR__>=((_maj)<<16)+(_min))
# else
# define OPUS_GNUC_PREREQ(_maj,_min) 0
# endif
# endif
#if (!defined(__STDC_VERSION__) || (__STDC_VERSION__ < 199901L) )
# if OPUS_GNUC_PREREQ(3,0)
# define OPUS_RESTRICT __restrict__
# elif (defined(_MSC_VER) && _MSC_VER >= 1400)
# define OPUS_RESTRICT __restrict
# else
# define OPUS_RESTRICT
# endif
#else
# define OPUS_RESTRICT restrict
#endif
#if (!defined(__STDC_VERSION__) || (__STDC_VERSION__ < 199901L) )
# if OPUS_GNUC_PREREQ(2,7)
# define OPUS_INLINE __inline__
# elif (defined(_MSC_VER))
# define OPUS_INLINE __inline
# else
# define OPUS_INLINE
# endif
#else
# define OPUS_INLINE inline
#endif
/**Warning attributes for opus functions
* NONNULL is not used in OPUS_BUILD to avoid the compiler optimizing out
* some paranoid null checks. */
#if defined(__GNUC__) && OPUS_GNUC_PREREQ(3, 4)
# define OPUS_WARN_UNUSED_RESULT __attribute__ ((__warn_unused_result__))
#else
# define OPUS_WARN_UNUSED_RESULT
#endif
#if !defined(OPUS_BUILD) && defined(__GNUC__) && OPUS_GNUC_PREREQ(3, 4)
# define OPUS_ARG_NONNULL(_x) __attribute__ ((__nonnull__(_x)))
#else
# define OPUS_ARG_NONNULL(_x)
#endif
/** These are the actual Encoder CTL ID numbers.
* They should not be used directly by applications.
* In general, SETs should be even and GETs should be odd.*/
#define OPUS_SET_APPLICATION_REQUEST 4000
#define OPUS_GET_APPLICATION_REQUEST 4001
#define OPUS_SET_BITRATE_REQUEST 4002
#define OPUS_GET_BITRATE_REQUEST 4003
#define OPUS_SET_MAX_BANDWIDTH_REQUEST 4004
#define OPUS_GET_MAX_BANDWIDTH_REQUEST 4005
#define OPUS_SET_VBR_REQUEST 4006
#define OPUS_GET_VBR_REQUEST 4007
#define OPUS_SET_BANDWIDTH_REQUEST 4008
#define OPUS_GET_BANDWIDTH_REQUEST 4009
#define OPUS_SET_COMPLEXITY_REQUEST 4010
#define OPUS_GET_COMPLEXITY_REQUEST 4011
#define OPUS_SET_INBAND_FEC_REQUEST 4012
#define OPUS_GET_INBAND_FEC_REQUEST 4013
#define OPUS_SET_PACKET_LOSS_PERC_REQUEST 4014
#define OPUS_GET_PACKET_LOSS_PERC_REQUEST 4015
#define OPUS_SET_DTX_REQUEST 4016
#define OPUS_GET_DTX_REQUEST 4017
#define OPUS_SET_VBR_CONSTRAINT_REQUEST 4020
#define OPUS_GET_VBR_CONSTRAINT_REQUEST 4021
#define OPUS_SET_FORCE_CHANNELS_REQUEST 4022
#define OPUS_GET_FORCE_CHANNELS_REQUEST 4023
#define OPUS_SET_SIGNAL_REQUEST 4024
#define OPUS_GET_SIGNAL_REQUEST 4025
#define OPUS_GET_LOOKAHEAD_REQUEST 4027
/* #define OPUS_RESET_STATE 4028 */
#define OPUS_GET_SAMPLE_RATE_REQUEST 4029
#define OPUS_GET_FINAL_RANGE_REQUEST 4031
#define OPUS_GET_PITCH_REQUEST 4033
#define OPUS_SET_GAIN_REQUEST 4034
#define OPUS_GET_GAIN_REQUEST 4045 /* Should have been 4035 */
#define OPUS_SET_LSB_DEPTH_REQUEST 4036
#define OPUS_GET_LSB_DEPTH_REQUEST 4037
#define OPUS_GET_LAST_PACKET_DURATION_REQUEST 4039
#define OPUS_SET_EXPERT_FRAME_DURATION_REQUEST 4040
#define OPUS_GET_EXPERT_FRAME_DURATION_REQUEST 4041
#define OPUS_SET_PREDICTION_DISABLED_REQUEST 4042
#define OPUS_GET_PREDICTION_DISABLED_REQUEST 4043
/* Don't use 4045, it's already taken by OPUS_GET_GAIN_REQUEST */
#define OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST 4046
#define OPUS_GET_PHASE_INVERSION_DISABLED_REQUEST 4047
#define OPUS_GET_IN_DTX_REQUEST 4049
/** Defines for the presence of extended APIs. */
#define OPUS_HAVE_OPUS_PROJECTION_H
/* Macros to trigger compilation errors when the wrong types are provided to a CTL */
#define __opus_check_int(x) (((void)((x) == (opus_int32)0)), (opus_int32)(x))
#define __opus_check_int_ptr(ptr) ((ptr) + ((ptr) - (opus_int32*)(ptr)))
#define __opus_check_uint_ptr(ptr) ((ptr) + ((ptr) - (opus_uint32*)(ptr)))
#define __opus_check_val16_ptr(ptr) ((ptr) + ((ptr) - (opus_val16*)(ptr)))
/** @endcond */
/** @defgroup opus_ctlvalues Pre-defined values for CTL interface
* @see opus_genericctls, opus_encoderctls
* @{
*/
/* Values for the various encoder CTLs */
#define OPUS_AUTO -1000 /**opus_int32: Allowed values: 0-10, inclusive.
*
* @hideinitializer */
#define OPUS_SET_COMPLEXITY(x) OPUS_SET_COMPLEXITY_REQUEST, __opus_check_int(x)
/** Gets the encoder's complexity configuration.
* @see OPUS_SET_COMPLEXITY
* @param[out] x opus_int32 *: Returns a value in the range 0-10,
* inclusive.
* @hideinitializer */
#define OPUS_GET_COMPLEXITY(x) OPUS_GET_COMPLEXITY_REQUEST, __opus_check_int_ptr(x)
/** Configures the bitrate in the encoder.
* Rates from 500 to 512000 bits per second are meaningful, as well as the
* special values #OPUS_AUTO and #OPUS_BITRATE_MAX.
* The value #OPUS_BITRATE_MAX can be used to cause the codec to use as much
* rate as it can, which is useful for controlling the rate by adjusting the
* output buffer size.
* @see OPUS_GET_BITRATE
* @param[in] x opus_int32: Bitrate in bits per second. The default
* is determined based on the number of
* channels and the input sampling rate.
* @hideinitializer */
#define OPUS_SET_BITRATE(x) OPUS_SET_BITRATE_REQUEST, __opus_check_int(x)
/** Gets the encoder's bitrate configuration.
* @see OPUS_SET_BITRATE
* @param[out] x opus_int32 *: Returns the bitrate in bits per second.
* The default is determined based on the
* number of channels and the input
* sampling rate.
* @hideinitializer */
#define OPUS_GET_BITRATE(x) OPUS_GET_BITRATE_REQUEST, __opus_check_int_ptr(x)
/** Enables or disables variable bitrate (VBR) in the encoder.
* The configured bitrate may not be met exactly because frames must
* be an integer number of bytes in length.
* @see OPUS_GET_VBR
* @see OPUS_SET_VBR_CONSTRAINT
* @param[in] x opus_int32: Allowed values:
*
*
0
Hard CBR. For LPC/hybrid modes at very low bit-rate, this can
* cause noticeable quality degradation.
*
1
VBR (default). The exact type of VBR is controlled by
* #OPUS_SET_VBR_CONSTRAINT.
*
* @hideinitializer */
#define OPUS_SET_VBR(x) OPUS_SET_VBR_REQUEST, __opus_check_int(x)
/** Determine if variable bitrate (VBR) is enabled in the encoder.
* @see OPUS_SET_VBR
* @see OPUS_GET_VBR_CONSTRAINT
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
0
Hard CBR.
*
1
VBR (default). The exact type of VBR may be retrieved via
* #OPUS_GET_VBR_CONSTRAINT.
*
* @hideinitializer */
#define OPUS_GET_VBR(x) OPUS_GET_VBR_REQUEST, __opus_check_int_ptr(x)
/** Enables or disables constrained VBR in the encoder.
* This setting is ignored when the encoder is in CBR mode.
* @warning Only the MDCT mode of Opus currently heeds the constraint.
* Speech mode ignores it completely, hybrid mode may fail to obey it
* if the LPC layer uses more bitrate than the constraint would have
* permitted.
* @see OPUS_GET_VBR_CONSTRAINT
* @see OPUS_SET_VBR
* @param[in] x opus_int32: Allowed values:
*
*
0
Unconstrained VBR.
*
1
Constrained VBR (default). This creates a maximum of one
* frame of buffering delay assuming a transport with a
* serialization speed of the nominal bitrate.
*
* @hideinitializer */
#define OPUS_SET_VBR_CONSTRAINT(x) OPUS_SET_VBR_CONSTRAINT_REQUEST, __opus_check_int(x)
/** Determine if constrained VBR is enabled in the encoder.
* @see OPUS_SET_VBR_CONSTRAINT
* @see OPUS_GET_VBR
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
0
Unconstrained VBR.
*
1
Constrained VBR (default).
*
* @hideinitializer */
#define OPUS_GET_VBR_CONSTRAINT(x) OPUS_GET_VBR_CONSTRAINT_REQUEST, __opus_check_int_ptr(x)
/** Configures mono/stereo forcing in the encoder.
* This can force the encoder to produce packets encoded as either mono or
* stereo, regardless of the format of the input audio. This is useful when
* the caller knows that the input signal is currently a mono source embedded
* in a stereo stream.
* @see OPUS_GET_FORCE_CHANNELS
* @param[in] x opus_int32: Allowed values:
*
* @hideinitializer */
#define OPUS_GET_FORCE_CHANNELS(x) OPUS_GET_FORCE_CHANNELS_REQUEST, __opus_check_int_ptr(x)
/** Configures the maximum bandpass that the encoder will select automatically.
* Applications should normally use this instead of #OPUS_SET_BANDWIDTH
* (leaving that set to the default, #OPUS_AUTO). This allows the
* application to set an upper bound based on the type of input it is
* providing, but still gives the encoder the freedom to reduce the bandpass
* when the bitrate becomes too low, for better overall quality.
* @see OPUS_GET_MAX_BANDWIDTH
* @param[in] x opus_int32: Allowed values:
*
*
OPUS_BANDWIDTH_NARROWBAND
4 kHz passband
*
OPUS_BANDWIDTH_MEDIUMBAND
6 kHz passband
*
OPUS_BANDWIDTH_WIDEBAND
8 kHz passband
*
OPUS_BANDWIDTH_SUPERWIDEBAND
12 kHz passband
*
OPUS_BANDWIDTH_FULLBAND
20 kHz passband (default)
*
* @hideinitializer */
#define OPUS_SET_MAX_BANDWIDTH(x) OPUS_SET_MAX_BANDWIDTH_REQUEST, __opus_check_int(x)
/** Gets the encoder's configured maximum allowed bandpass.
* @see OPUS_SET_MAX_BANDWIDTH
* @param[out] x opus_int32 *: Allowed values:
*
*
#OPUS_BANDWIDTH_NARROWBAND
4 kHz passband
*
#OPUS_BANDWIDTH_MEDIUMBAND
6 kHz passband
*
#OPUS_BANDWIDTH_WIDEBAND
8 kHz passband
*
#OPUS_BANDWIDTH_SUPERWIDEBAND
12 kHz passband
*
#OPUS_BANDWIDTH_FULLBAND
20 kHz passband (default)
*
* @hideinitializer */
#define OPUS_GET_MAX_BANDWIDTH(x) OPUS_GET_MAX_BANDWIDTH_REQUEST, __opus_check_int_ptr(x)
/** Sets the encoder's bandpass to a specific value.
* This prevents the encoder from automatically selecting the bandpass based
* on the available bitrate. If an application knows the bandpass of the input
* audio it is providing, it should normally use #OPUS_SET_MAX_BANDWIDTH
* instead, which still gives the encoder the freedom to reduce the bandpass
* when the bitrate becomes too low, for better overall quality.
* @see OPUS_GET_BANDWIDTH
* @param[in] x opus_int32: Allowed values:
*
*
#OPUS_AUTO
(default)
*
#OPUS_BANDWIDTH_NARROWBAND
4 kHz passband
*
#OPUS_BANDWIDTH_MEDIUMBAND
6 kHz passband
*
#OPUS_BANDWIDTH_WIDEBAND
8 kHz passband
*
#OPUS_BANDWIDTH_SUPERWIDEBAND
12 kHz passband
*
#OPUS_BANDWIDTH_FULLBAND
20 kHz passband
*
* @hideinitializer */
#define OPUS_SET_BANDWIDTH(x) OPUS_SET_BANDWIDTH_REQUEST, __opus_check_int(x)
/** Configures the type of signal being encoded.
* This is a hint which helps the encoder's mode selection.
* @see OPUS_GET_SIGNAL
* @param[in] x opus_int32: Allowed values:
*
*
#OPUS_AUTO
(default)
*
#OPUS_SIGNAL_VOICE
Bias thresholds towards choosing LPC or Hybrid modes.
*
#OPUS_SIGNAL_MUSIC
Bias thresholds towards choosing MDCT modes.
*
* @hideinitializer */
#define OPUS_SET_SIGNAL(x) OPUS_SET_SIGNAL_REQUEST, __opus_check_int(x)
/** Gets the encoder's configured signal type.
* @see OPUS_SET_SIGNAL
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
#OPUS_AUTO
(default)
*
#OPUS_SIGNAL_VOICE
Bias thresholds towards choosing LPC or Hybrid modes.
*
#OPUS_SIGNAL_MUSIC
Bias thresholds towards choosing MDCT modes.
*
* @hideinitializer */
#define OPUS_GET_SIGNAL(x) OPUS_GET_SIGNAL_REQUEST, __opus_check_int_ptr(x)
/** Configures the encoder's intended application.
* The initial value is a mandatory argument to the encoder_create function.
* @see OPUS_GET_APPLICATION
* @param[in] x opus_int32: Returns one of the following values:
*
*
#OPUS_APPLICATION_VOIP
*
Process signal for improved speech intelligibility.
*
#OPUS_APPLICATION_AUDIO
*
Favor faithfulness to the original input.
*
#OPUS_APPLICATION_RESTRICTED_LOWDELAY
*
Configure the minimum possible coding delay by disabling certain modes
* of operation.
*
* @hideinitializer */
#define OPUS_SET_APPLICATION(x) OPUS_SET_APPLICATION_REQUEST, __opus_check_int(x)
/** Gets the encoder's configured application.
* @see OPUS_SET_APPLICATION
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
#OPUS_APPLICATION_VOIP
*
Process signal for improved speech intelligibility.
*
#OPUS_APPLICATION_AUDIO
*
Favor faithfulness to the original input.
*
#OPUS_APPLICATION_RESTRICTED_LOWDELAY
*
Configure the minimum possible coding delay by disabling certain modes
* of operation.
*
* @hideinitializer */
#define OPUS_GET_APPLICATION(x) OPUS_GET_APPLICATION_REQUEST, __opus_check_int_ptr(x)
/** Gets the total samples of delay added by the entire codec.
* This can be queried by the encoder and then the provided number of samples can be
* skipped on from the start of the decoder's output to provide time aligned input
* and output. From the perspective of a decoding application the real data begins this many
* samples late.
*
* The decoder contribution to this delay is identical for all decoders, but the
* encoder portion of the delay may vary from implementation to implementation,
* version to version, or even depend on the encoder's initial configuration.
* Applications needing delay compensation should call this CTL rather than
* hard-coding a value.
* @param[out] x opus_int32 *: Number of lookahead samples
* @hideinitializer */
#define OPUS_GET_LOOKAHEAD(x) OPUS_GET_LOOKAHEAD_REQUEST, __opus_check_int_ptr(x)
/** Configures the encoder's use of inband forward error correction (FEC).
* @note This is only applicable to the LPC layer
* @see OPUS_GET_INBAND_FEC
* @param[in] x opus_int32: Allowed values:
*
*
0
Disable inband FEC (default).
*
1
Enable inband FEC.
*
* @hideinitializer */
#define OPUS_SET_INBAND_FEC(x) OPUS_SET_INBAND_FEC_REQUEST, __opus_check_int(x)
/** Gets encoder's configured use of inband forward error correction.
* @see OPUS_SET_INBAND_FEC
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
0
Inband FEC disabled (default).
*
1
Inband FEC enabled.
*
* @hideinitializer */
#define OPUS_GET_INBAND_FEC(x) OPUS_GET_INBAND_FEC_REQUEST, __opus_check_int_ptr(x)
/** Configures the encoder's expected packet loss percentage.
* Higher values trigger progressively more loss resistant behavior in the encoder
* at the expense of quality at a given bitrate in the absence of packet loss, but
* greater quality under loss.
* @see OPUS_GET_PACKET_LOSS_PERC
* @param[in] x opus_int32: Loss percentage in the range 0-100, inclusive (default: 0).
* @hideinitializer */
#define OPUS_SET_PACKET_LOSS_PERC(x) OPUS_SET_PACKET_LOSS_PERC_REQUEST, __opus_check_int(x)
/** Gets the encoder's configured packet loss percentage.
* @see OPUS_SET_PACKET_LOSS_PERC
* @param[out] x opus_int32 *: Returns the configured loss percentage
* in the range 0-100, inclusive (default: 0).
* @hideinitializer */
#define OPUS_GET_PACKET_LOSS_PERC(x) OPUS_GET_PACKET_LOSS_PERC_REQUEST, __opus_check_int_ptr(x)
/** Configures the encoder's use of discontinuous transmission (DTX).
* @note This is only applicable to the LPC layer
* @see OPUS_GET_DTX
* @param[in] x opus_int32: Allowed values:
*
*
0
Disable DTX (default).
*
1
Enabled DTX.
*
* @hideinitializer */
#define OPUS_SET_DTX(x) OPUS_SET_DTX_REQUEST, __opus_check_int(x)
/** Gets encoder's configured use of discontinuous transmission.
* @see OPUS_SET_DTX
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
0
DTX disabled (default).
*
1
DTX enabled.
*
* @hideinitializer */
#define OPUS_GET_DTX(x) OPUS_GET_DTX_REQUEST, __opus_check_int_ptr(x)
/** Configures the depth of signal being encoded.
*
* This is a hint which helps the encoder identify silence and near-silence.
* It represents the number of significant bits of linear intensity below
* which the signal contains ignorable quantization or other noise.
*
* For example, OPUS_SET_LSB_DEPTH(14) would be an appropriate setting
* for G.711 u-law input. OPUS_SET_LSB_DEPTH(16) would be appropriate
* for 16-bit linear pcm input with opus_encode_float().
*
* When using opus_encode() instead of opus_encode_float(), or when libopus
* is compiled for fixed-point, the encoder uses the minimum of the value
* set here and the value 16.
*
* @see OPUS_GET_LSB_DEPTH
* @param[in] x opus_int32: Input precision in bits, between 8 and 24
* (default: 24).
* @hideinitializer */
#define OPUS_SET_LSB_DEPTH(x) OPUS_SET_LSB_DEPTH_REQUEST, __opus_check_int(x)
/** Gets the encoder's configured signal depth.
* @see OPUS_SET_LSB_DEPTH
* @param[out] x opus_int32 *: Input precision in bits, between 8 and
* 24 (default: 24).
* @hideinitializer */
#define OPUS_GET_LSB_DEPTH(x) OPUS_GET_LSB_DEPTH_REQUEST, __opus_check_int_ptr(x)
/** Configures the encoder's use of variable duration frames.
* When variable duration is enabled, the encoder is free to use a shorter frame
* size than the one requested in the opus_encode*() call.
* It is then the user's responsibility
* to verify how much audio was encoded by checking the ToC byte of the encoded
* packet. The part of the audio that was not encoded needs to be resent to the
* encoder for the next call. Do not use this option unless you really
* know what you are doing.
* @see OPUS_GET_EXPERT_FRAME_DURATION
* @param[in] x opus_int32: Allowed values:
*
*
OPUS_FRAMESIZE_ARG
Select frame size from the argument (default).
*
OPUS_FRAMESIZE_2_5_MS
Use 2.5 ms frames.
*
OPUS_FRAMESIZE_5_MS
Use 5 ms frames.
*
OPUS_FRAMESIZE_10_MS
Use 10 ms frames.
*
OPUS_FRAMESIZE_20_MS
Use 20 ms frames.
*
OPUS_FRAMESIZE_40_MS
Use 40 ms frames.
*
OPUS_FRAMESIZE_60_MS
Use 60 ms frames.
*
OPUS_FRAMESIZE_80_MS
Use 80 ms frames.
*
OPUS_FRAMESIZE_100_MS
Use 100 ms frames.
*
OPUS_FRAMESIZE_120_MS
Use 120 ms frames.
*
* @hideinitializer */
#define OPUS_SET_EXPERT_FRAME_DURATION(x) OPUS_SET_EXPERT_FRAME_DURATION_REQUEST, __opus_check_int(x)
/** Gets the encoder's configured use of variable duration frames.
* @see OPUS_SET_EXPERT_FRAME_DURATION
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
OPUS_FRAMESIZE_ARG
Select frame size from the argument (default).
*
OPUS_FRAMESIZE_2_5_MS
Use 2.5 ms frames.
*
OPUS_FRAMESIZE_5_MS
Use 5 ms frames.
*
OPUS_FRAMESIZE_10_MS
Use 10 ms frames.
*
OPUS_FRAMESIZE_20_MS
Use 20 ms frames.
*
OPUS_FRAMESIZE_40_MS
Use 40 ms frames.
*
OPUS_FRAMESIZE_60_MS
Use 60 ms frames.
*
OPUS_FRAMESIZE_80_MS
Use 80 ms frames.
*
OPUS_FRAMESIZE_100_MS
Use 100 ms frames.
*
OPUS_FRAMESIZE_120_MS
Use 120 ms frames.
*
* @hideinitializer */
#define OPUS_GET_EXPERT_FRAME_DURATION(x) OPUS_GET_EXPERT_FRAME_DURATION_REQUEST, __opus_check_int_ptr(x)
/** If set to 1, disables almost all use of prediction, making frames almost
* completely independent. This reduces quality.
* @see OPUS_GET_PREDICTION_DISABLED
* @param[in] x opus_int32: Allowed values:
*
*
0
Enable prediction (default).
*
1
Disable prediction.
*
* @hideinitializer */
#define OPUS_SET_PREDICTION_DISABLED(x) OPUS_SET_PREDICTION_DISABLED_REQUEST, __opus_check_int(x)
/** Gets the encoder's configured prediction status.
* @see OPUS_SET_PREDICTION_DISABLED
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
0
Prediction enabled (default).
*
1
Prediction disabled.
*
* @hideinitializer */
#define OPUS_GET_PREDICTION_DISABLED(x) OPUS_GET_PREDICTION_DISABLED_REQUEST, __opus_check_int_ptr(x)
/**@}*/
/** @defgroup opus_genericctls Generic CTLs
*
* These macros are used with the \c opus_decoder_ctl and
* \c opus_encoder_ctl calls to generate a particular
* request.
*
* When called on an \c OpusDecoder they apply to that
* particular decoder instance. When called on an
* \c OpusEncoder they apply to the corresponding setting
* on that encoder instance, if present.
*
* Some usage examples:
*
* @code
* int ret;
* opus_int32 pitch;
* ret = opus_decoder_ctl(dec_ctx, OPUS_GET_PITCH(&pitch));
* if (ret == OPUS_OK) return ret;
*
* opus_encoder_ctl(enc_ctx, OPUS_RESET_STATE);
* opus_decoder_ctl(dec_ctx, OPUS_RESET_STATE);
*
* opus_int32 enc_bw, dec_bw;
* opus_encoder_ctl(enc_ctx, OPUS_GET_BANDWIDTH(&enc_bw));
* opus_decoder_ctl(dec_ctx, OPUS_GET_BANDWIDTH(&dec_bw));
* if (enc_bw != dec_bw) {
* printf("packet bandwidth mismatch!\n");
* }
* @endcode
*
* @see opus_encoder, opus_decoder_ctl, opus_encoder_ctl, opus_decoderctls, opus_encoderctls
* @{
*/
/** Resets the codec state to be equivalent to a freshly initialized state.
* This should be called when switching streams in order to prevent
* the back to back decoding from giving different results from
* one at a time decoding.
* @hideinitializer */
#define OPUS_RESET_STATE 4028
/** Gets the final state of the codec's entropy coder.
* This is used for testing purposes,
* The encoder and decoder state should be identical after coding a payload
* (assuming no data corruption or software bugs)
*
* @param[out] x opus_uint32 *: Entropy coder state
*
* @hideinitializer */
#define OPUS_GET_FINAL_RANGE(x) OPUS_GET_FINAL_RANGE_REQUEST, __opus_check_uint_ptr(x)
/** Gets the encoder's configured bandpass or the decoder's last bandpass.
* @see OPUS_SET_BANDWIDTH
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
#OPUS_AUTO
(default)
*
#OPUS_BANDWIDTH_NARROWBAND
4 kHz passband
*
#OPUS_BANDWIDTH_MEDIUMBAND
6 kHz passband
*
#OPUS_BANDWIDTH_WIDEBAND
8 kHz passband
*
#OPUS_BANDWIDTH_SUPERWIDEBAND
12 kHz passband
*
#OPUS_BANDWIDTH_FULLBAND
20 kHz passband
*
* @hideinitializer */
#define OPUS_GET_BANDWIDTH(x) OPUS_GET_BANDWIDTH_REQUEST, __opus_check_int_ptr(x)
/** Gets the sampling rate the encoder or decoder was initialized with.
* This simply returns the Fs value passed to opus_encoder_init()
* or opus_decoder_init().
* @param[out] x opus_int32 *: Sampling rate of encoder or decoder.
* @hideinitializer
*/
#define OPUS_GET_SAMPLE_RATE(x) OPUS_GET_SAMPLE_RATE_REQUEST, __opus_check_int_ptr(x)
/** If set to 1, disables the use of phase inversion for intensity stereo,
* improving the quality of mono downmixes, but slightly reducing normal
* stereo quality. Disabling phase inversion in the decoder does not comply
* with RFC 6716, although it does not cause any interoperability issue and
* is expected to become part of the Opus standard once RFC 6716 is updated
* by draft-ietf-codec-opus-update.
* @see OPUS_GET_PHASE_INVERSION_DISABLED
* @param[in] x opus_int32: Allowed values:
*
*
0
Enable phase inversion (default).
*
1
Disable phase inversion.
*
* @hideinitializer */
#define OPUS_SET_PHASE_INVERSION_DISABLED(x) OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST, __opus_check_int(x)
/** Gets the encoder's configured phase inversion status.
* @see OPUS_SET_PHASE_INVERSION_DISABLED
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
0
Stereo phase inversion enabled (default).
*
1
Stereo phase inversion disabled.
*
* @hideinitializer */
#define OPUS_GET_PHASE_INVERSION_DISABLED(x) OPUS_GET_PHASE_INVERSION_DISABLED_REQUEST, __opus_check_int_ptr(x)
/** Gets the DTX state of the encoder.
* Returns whether the last encoded frame was either a comfort noise update
* during DTX or not encoded because of DTX.
* @param[out] x opus_int32 *: Returns one of the following values:
*
*
0
The encoder is not in DTX.
*
1
The encoder is in DTX.
*
* @hideinitializer */
#define OPUS_GET_IN_DTX(x) OPUS_GET_IN_DTX_REQUEST, __opus_check_int_ptr(x)
/**@}*/
/** @defgroup opus_decoderctls Decoder related CTLs
* @see opus_genericctls, opus_encoderctls, opus_decoder
* @{
*/
/** Configures decoder gain adjustment.
* Scales the decoded output by a factor specified in Q8 dB units.
* This has a maximum range of -32768 to 32767 inclusive, and returns
* OPUS_BAD_ARG otherwise. The default is zero indicating no adjustment.
* This setting survives decoder reset.
*
* gain = pow(10, x/(20.0*256))
*
* @param[in] x opus_int32: Amount to scale PCM signal by in Q8 dB units.
* @hideinitializer */
#define OPUS_SET_GAIN(x) OPUS_SET_GAIN_REQUEST, __opus_check_int(x)
/** Gets the decoder's configured gain adjustment. @see OPUS_SET_GAIN
*
* @param[out] x opus_int32 *: Amount to scale PCM signal by in Q8 dB units.
* @hideinitializer */
#define OPUS_GET_GAIN(x) OPUS_GET_GAIN_REQUEST, __opus_check_int_ptr(x)
/** Gets the duration (in samples) of the last packet successfully decoded or concealed.
* @param[out] x opus_int32 *: Number of samples (at current sampling rate).
* @hideinitializer */
#define OPUS_GET_LAST_PACKET_DURATION(x) OPUS_GET_LAST_PACKET_DURATION_REQUEST, __opus_check_int_ptr(x)
/** Gets the pitch of the last decoded frame, if available.
* This can be used for any post-processing algorithm requiring the use of pitch,
* e.g. time stretching/shortening. If the last frame was not voiced, or if the
* pitch was not coded in the frame, then zero is returned.
*
* This CTL is only implemented for decoder instances.
*
* @param[out] x opus_int32 *: pitch period at 48 kHz (or 0 if not available)
*
* @hideinitializer */
#define OPUS_GET_PITCH(x) OPUS_GET_PITCH_REQUEST, __opus_check_int_ptr(x)
/**@}*/
/** @defgroup opus_libinfo Opus library information functions
* @{
*/
/** Converts an opus error code into a human readable string.
*
* @param[in] error int: Error number
* @returns Error string
*/
OPUS_EXPORT const char *opus_strerror(int error);
/** Gets the libopus version string.
*
* Applications may look for the substring "-fixed" in the version string to
* determine whether they have a fixed-point or floating-point build at
* runtime.
*
* @returns Version string
*/
OPUS_EXPORT const char *opus_get_version_string(void);
/**@}*/
#ifdef __cplusplus
}
#endif
#endif /* OPUS_DEFINES_H */
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libopus/opus_multistream.h
================================================
/* Copyright (c) 2011 Xiph.Org Foundation
Written by Jean-Marc Valin */
/*
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/**
* @file opus_multistream.h
* @brief Opus reference implementation multistream API
*/
#ifndef OPUS_MULTISTREAM_H
#define OPUS_MULTISTREAM_H
#include "opus.h"
#ifdef __cplusplus
extern "C" {
#endif
/** @cond OPUS_INTERNAL_DOC */
/** Macros to trigger compilation errors when the wrong types are provided to a
* CTL. */
/**@{*/
#define __opus_check_encstate_ptr(ptr) ((ptr) + ((ptr) - (OpusEncoder**)(ptr)))
#define __opus_check_decstate_ptr(ptr) ((ptr) + ((ptr) - (OpusDecoder**)(ptr)))
/**@}*/
/** These are the actual encoder and decoder CTL ID numbers.
* They should not be used directly by applications.
* In general, SETs should be even and GETs should be odd.*/
/**@{*/
#define OPUS_MULTISTREAM_GET_ENCODER_STATE_REQUEST 5120
#define OPUS_MULTISTREAM_GET_DECODER_STATE_REQUEST 5122
/**@}*/
/** @endcond */
/** @defgroup opus_multistream_ctls Multistream specific encoder and decoder CTLs
*
* These are convenience macros that are specific to the
* opus_multistream_encoder_ctl() and opus_multistream_decoder_ctl()
* interface.
* The CTLs from @ref opus_genericctls, @ref opus_encoderctls, and
* @ref opus_decoderctls may be applied to a multistream encoder or decoder as
* well.
* In addition, you may retrieve the encoder or decoder state for an specific
* stream via #OPUS_MULTISTREAM_GET_ENCODER_STATE or
* #OPUS_MULTISTREAM_GET_DECODER_STATE and apply CTLs to it individually.
*/
/**@{*/
/** Gets the encoder state for an individual stream of a multistream encoder.
* @param[in] x opus_int32: The index of the stream whose encoder you
* wish to retrieve.
* This must be non-negative and less than
* the streams parameter used
* to initialize the encoder.
* @param[out] y OpusEncoder**: Returns a pointer to the given
* encoder state.
* @retval OPUS_BAD_ARG The index of the requested stream was out of range.
* @hideinitializer
*/
#define OPUS_MULTISTREAM_GET_ENCODER_STATE(x,y) OPUS_MULTISTREAM_GET_ENCODER_STATE_REQUEST, __opus_check_int(x), __opus_check_encstate_ptr(y)
/** Gets the decoder state for an individual stream of a multistream decoder.
* @param[in] x opus_int32: The index of the stream whose decoder you
* wish to retrieve.
* This must be non-negative and less than
* the streams parameter used
* to initialize the decoder.
* @param[out] y OpusDecoder**: Returns a pointer to the given
* decoder state.
* @retval OPUS_BAD_ARG The index of the requested stream was out of range.
* @hideinitializer
*/
#define OPUS_MULTISTREAM_GET_DECODER_STATE(x,y) OPUS_MULTISTREAM_GET_DECODER_STATE_REQUEST, __opus_check_int(x), __opus_check_decstate_ptr(y)
/**@}*/
/** @defgroup opus_multistream Opus Multistream API
* @{
*
* The multistream API allows individual Opus streams to be combined into a
* single packet, enabling support for up to 255 channels. Unlike an
* elementary Opus stream, the encoder and decoder must negotiate the channel
* configuration before the decoder can successfully interpret the data in the
* packets produced by the encoder. Some basic information, such as packet
* duration, can be computed without any special negotiation.
*
* The format for multistream Opus packets is defined in
* RFC 7845
* and is based on the self-delimited Opus framing described in Appendix B of
* RFC 6716.
* Normal Opus packets are just a degenerate case of multistream Opus packets,
* and can be encoded or decoded with the multistream API by setting
* streams to 1 when initializing the encoder or
* decoder.
*
* Multistream Opus streams can contain up to 255 elementary Opus streams.
* These may be either "uncoupled" or "coupled", indicating that the decoder
* is configured to decode them to either 1 or 2 channels, respectively.
* The streams are ordered so that all coupled streams appear at the
* beginning.
*
* A mapping table defines which decoded channel i
* should be used for each input/output (I/O) channel j. This table is
* typically provided as an unsigned char array.
* Let i = mapping[j] be the index for I/O channel j.
* If i < 2*coupled_streams, then I/O channel j is
* encoded as the left channel of stream (i/2) if i
* is even, or as the right channel of stream (i/2) if
* i is odd. Otherwise, I/O channel j is encoded as
* mono in stream (i - coupled_streams), unless it has the special
* value 255, in which case it is omitted from the encoding entirely (the
* decoder will reproduce it as silence). Each value i must either
* be the special value 255 or be less than streams + coupled_streams.
*
* The output channels specified by the encoder
* should use the
* Vorbis
* channel ordering. A decoder may wish to apply an additional permutation
* to the mapping the encoder used to achieve a different output channel
* order (e.g. for outputing in WAV order).
*
* Each multistream packet contains an Opus packet for each stream, and all of
* the Opus packets in a single multistream packet must have the same
* duration. Therefore the duration of a multistream packet can be extracted
* from the TOC sequence of the first stream, which is located at the
* beginning of the packet, just like an elementary Opus stream:
*
* @code
* int nb_samples;
* int nb_frames;
* nb_frames = opus_packet_get_nb_frames(data, len);
* if (nb_frames < 1)
* return nb_frames;
* nb_samples = opus_packet_get_samples_per_frame(data, 48000) * nb_frames;
* @endcode
*
* The general encoding and decoding process proceeds exactly the same as in
* the normal @ref opus_encoder and @ref opus_decoder APIs.
* See their documentation for an overview of how to use the corresponding
* multistream functions.
*/
/** Opus multistream encoder state.
* This contains the complete state of a multistream Opus encoder.
* It is position independent and can be freely copied.
* @see opus_multistream_encoder_create
* @see opus_multistream_encoder_init
*/
typedef struct OpusMSEncoder OpusMSEncoder;
/** Opus multistream decoder state.
* This contains the complete state of a multistream Opus decoder.
* It is position independent and can be freely copied.
* @see opus_multistream_decoder_create
* @see opus_multistream_decoder_init
*/
typedef struct OpusMSDecoder OpusMSDecoder;
/**\name Multistream encoder functions */
/**@{*/
/** Gets the size of an OpusMSEncoder structure.
* @param streams int: The total number of streams to encode from the
* input.
* This must be no more than 255.
* @param coupled_streams int: Number of coupled (2 channel) streams
* to encode.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* encoded channels (streams +
* coupled_streams) must be no
* more than 255.
* @returns The size in bytes on success, or a negative error code
* (see @ref opus_errorcodes) on error.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_multistream_encoder_get_size(
int streams,
int coupled_streams
);
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_multistream_surround_encoder_get_size(
int channels,
int mapping_family
);
/** Allocates and initializes a multistream encoder state.
* Call opus_multistream_encoder_destroy() to release
* this object when finished.
* @param Fs opus_int32: Sampling rate of the input signal (in Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param channels int: Number of channels in the input signal.
* This must be at most 255.
* It may be greater than the number of
* coded channels (streams +
* coupled_streams).
* @param streams int: The total number of streams to encode from the
* input.
* This must be no more than the number of channels.
* @param coupled_streams int: Number of coupled (2 channel) streams
* to encode.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* encoded channels (streams +
* coupled_streams) must be no
* more than the number of input channels.
* @param[in] mapping const unsigned char[channels]: Mapping from
* encoded channels to input channels, as described in
* @ref opus_multistream. As an extra constraint, the
* multistream encoder does not allow encoding coupled
* streams for which one channel is unused since this
* is never a good idea.
* @param application int: The target encoder application.
* This must be one of the following:
*
*
#OPUS_APPLICATION_VOIP
*
Process signal for improved speech intelligibility.
*
#OPUS_APPLICATION_AUDIO
*
Favor faithfulness to the original input.
*
#OPUS_APPLICATION_RESTRICTED_LOWDELAY
*
Configure the minimum possible coding delay by disabling certain modes
* of operation.
*
* @param[out] error int *: Returns #OPUS_OK on success, or an error
* code (see @ref opus_errorcodes) on
* failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT OpusMSEncoder *opus_multistream_encoder_create(
opus_int32 Fs,
int channels,
int streams,
int coupled_streams,
const unsigned char *mapping,
int application,
int *error
) OPUS_ARG_NONNULL(5);
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT OpusMSEncoder *opus_multistream_surround_encoder_create(
opus_int32 Fs,
int channels,
int mapping_family,
int *streams,
int *coupled_streams,
unsigned char *mapping,
int application,
int *error
) OPUS_ARG_NONNULL(4) OPUS_ARG_NONNULL(5) OPUS_ARG_NONNULL(6);
/** Initialize a previously allocated multistream encoder state.
* The memory pointed to by \a st must be at least the size returned by
* opus_multistream_encoder_get_size().
* This is intended for applications which use their own allocator instead of
* malloc.
* To reset a previously initialized state, use the #OPUS_RESET_STATE CTL.
* @see opus_multistream_encoder_create
* @see opus_multistream_encoder_get_size
* @param st OpusMSEncoder*: Multistream encoder state to initialize.
* @param Fs opus_int32: Sampling rate of the input signal (in Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param channels int: Number of channels in the input signal.
* This must be at most 255.
* It may be greater than the number of
* coded channels (streams +
* coupled_streams).
* @param streams int: The total number of streams to encode from the
* input.
* This must be no more than the number of channels.
* @param coupled_streams int: Number of coupled (2 channel) streams
* to encode.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* encoded channels (streams +
* coupled_streams) must be no
* more than the number of input channels.
* @param[in] mapping const unsigned char[channels]: Mapping from
* encoded channels to input channels, as described in
* @ref opus_multistream. As an extra constraint, the
* multistream encoder does not allow encoding coupled
* streams for which one channel is unused since this
* is never a good idea.
* @param application int: The target encoder application.
* This must be one of the following:
*
*
#OPUS_APPLICATION_VOIP
*
Process signal for improved speech intelligibility.
*
#OPUS_APPLICATION_AUDIO
*
Favor faithfulness to the original input.
*
#OPUS_APPLICATION_RESTRICTED_LOWDELAY
*
Configure the minimum possible coding delay by disabling certain modes
* of operation.
*
* @returns #OPUS_OK on success, or an error code (see @ref opus_errorcodes)
* on failure.
*/
OPUS_EXPORT int opus_multistream_encoder_init(
OpusMSEncoder *st,
opus_int32 Fs,
int channels,
int streams,
int coupled_streams,
const unsigned char *mapping,
int application
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(6);
OPUS_EXPORT int opus_multistream_surround_encoder_init(
OpusMSEncoder *st,
opus_int32 Fs,
int channels,
int mapping_family,
int *streams,
int *coupled_streams,
unsigned char *mapping,
int application
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(5) OPUS_ARG_NONNULL(6) OPUS_ARG_NONNULL(7);
/** Encodes a multistream Opus frame.
* @param st OpusMSEncoder*: Multistream encoder state.
* @param[in] pcm const opus_int16*: The input signal as interleaved
* samples.
* This must contain
* frame_size*channels
* samples.
* @param frame_size int: Number of samples per channel in the input
* signal.
* This must be an Opus frame size for the
* encoder's sampling rate.
* For example, at 48 kHz the permitted values
* are 120, 240, 480, 960, 1920, and 2880.
* Passing in a duration of less than 10 ms
* (480 samples at 48 kHz) will prevent the
* encoder from using the LPC or hybrid modes.
* @param[out] data unsigned char*: Output payload.
* This must contain storage for at
* least \a max_data_bytes.
* @param [in] max_data_bytes opus_int32: Size of the allocated
* memory for the output
* payload. This may be
* used to impose an upper limit on
* the instant bitrate, but should
* not be used as the only bitrate
* control. Use #OPUS_SET_BITRATE to
* control the bitrate.
* @returns The length of the encoded packet (in bytes) on success or a
* negative error code (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_multistream_encode(
OpusMSEncoder *st,
const opus_int16 *pcm,
int frame_size,
unsigned char *data,
opus_int32 max_data_bytes
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2) OPUS_ARG_NONNULL(4);
/** Encodes a multistream Opus frame from floating point input.
* @param st OpusMSEncoder*: Multistream encoder state.
* @param[in] pcm const float*: The input signal as interleaved
* samples with a normal range of
* +/-1.0.
* Samples with a range beyond +/-1.0
* are supported but will be clipped by
* decoders using the integer API and
* should only be used if it is known
* that the far end supports extended
* dynamic range.
* This must contain
* frame_size*channels
* samples.
* @param frame_size int: Number of samples per channel in the input
* signal.
* This must be an Opus frame size for the
* encoder's sampling rate.
* For example, at 48 kHz the permitted values
* are 120, 240, 480, 960, 1920, and 2880.
* Passing in a duration of less than 10 ms
* (480 samples at 48 kHz) will prevent the
* encoder from using the LPC or hybrid modes.
* @param[out] data unsigned char*: Output payload.
* This must contain storage for at
* least \a max_data_bytes.
* @param [in] max_data_bytes opus_int32: Size of the allocated
* memory for the output
* payload. This may be
* used to impose an upper limit on
* the instant bitrate, but should
* not be used as the only bitrate
* control. Use #OPUS_SET_BITRATE to
* control the bitrate.
* @returns The length of the encoded packet (in bytes) on success or a
* negative error code (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_multistream_encode_float(
OpusMSEncoder *st,
const float *pcm,
int frame_size,
unsigned char *data,
opus_int32 max_data_bytes
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2) OPUS_ARG_NONNULL(4);
/** Frees an OpusMSEncoder allocated by
* opus_multistream_encoder_create().
* @param st OpusMSEncoder*: Multistream encoder state to be freed.
*/
OPUS_EXPORT void opus_multistream_encoder_destroy(OpusMSEncoder *st);
/** Perform a CTL function on a multistream Opus encoder.
*
* Generally the request and subsequent arguments are generated by a
* convenience macro.
* @param st OpusMSEncoder*: Multistream encoder state.
* @param request This and all remaining parameters should be replaced by one
* of the convenience macros in @ref opus_genericctls,
* @ref opus_encoderctls, or @ref opus_multistream_ctls.
* @see opus_genericctls
* @see opus_encoderctls
* @see opus_multistream_ctls
*/
OPUS_EXPORT int opus_multistream_encoder_ctl(OpusMSEncoder *st, int request, ...) OPUS_ARG_NONNULL(1);
/**@}*/
/**\name Multistream decoder functions */
/**@{*/
/** Gets the size of an OpusMSDecoder structure.
* @param streams int: The total number of streams coded in the
* input.
* This must be no more than 255.
* @param coupled_streams int: Number streams to decode as coupled
* (2 channel) streams.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* coded channels (streams +
* coupled_streams) must be no
* more than 255.
* @returns The size in bytes on success, or a negative error code
* (see @ref opus_errorcodes) on error.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_multistream_decoder_get_size(
int streams,
int coupled_streams
);
/** Allocates and initializes a multistream decoder state.
* Call opus_multistream_decoder_destroy() to release
* this object when finished.
* @param Fs opus_int32: Sampling rate to decode at (in Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param channels int: Number of channels to output.
* This must be at most 255.
* It may be different from the number of coded
* channels (streams +
* coupled_streams).
* @param streams int: The total number of streams coded in the
* input.
* This must be no more than 255.
* @param coupled_streams int: Number of streams to decode as coupled
* (2 channel) streams.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* coded channels (streams +
* coupled_streams) must be no
* more than 255.
* @param[in] mapping const unsigned char[channels]: Mapping from
* coded channels to output channels, as described in
* @ref opus_multistream.
* @param[out] error int *: Returns #OPUS_OK on success, or an error
* code (see @ref opus_errorcodes) on
* failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT OpusMSDecoder *opus_multistream_decoder_create(
opus_int32 Fs,
int channels,
int streams,
int coupled_streams,
const unsigned char *mapping,
int *error
) OPUS_ARG_NONNULL(5);
/** Intialize a previously allocated decoder state object.
* The memory pointed to by \a st must be at least the size returned by
* opus_multistream_encoder_get_size().
* This is intended for applications which use their own allocator instead of
* malloc.
* To reset a previously initialized state, use the #OPUS_RESET_STATE CTL.
* @see opus_multistream_decoder_create
* @see opus_multistream_deocder_get_size
* @param st OpusMSEncoder*: Multistream encoder state to initialize.
* @param Fs opus_int32: Sampling rate to decode at (in Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param channels int: Number of channels to output.
* This must be at most 255.
* It may be different from the number of coded
* channels (streams +
* coupled_streams).
* @param streams int: The total number of streams coded in the
* input.
* This must be no more than 255.
* @param coupled_streams int: Number of streams to decode as coupled
* (2 channel) streams.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* coded channels (streams +
* coupled_streams) must be no
* more than 255.
* @param[in] mapping const unsigned char[channels]: Mapping from
* coded channels to output channels, as described in
* @ref opus_multistream.
* @returns #OPUS_OK on success, or an error code (see @ref opus_errorcodes)
* on failure.
*/
OPUS_EXPORT int opus_multistream_decoder_init(
OpusMSDecoder *st,
opus_int32 Fs,
int channels,
int streams,
int coupled_streams,
const unsigned char *mapping
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(6);
/** Decode a multistream Opus packet.
* @param st OpusMSDecoder*: Multistream decoder state.
* @param[in] data const unsigned char*: Input payload.
* Use a NULL
* pointer to indicate packet
* loss.
* @param len opus_int32: Number of bytes in payload.
* @param[out] pcm opus_int16*: Output signal, with interleaved
* samples.
* This must contain room for
* frame_size*channels
* samples.
* @param frame_size int: The number of samples per channel of
* available space in \a pcm.
* If this is less than the maximum packet duration
* (120 ms; 5760 for 48kHz), this function will not be capable
* of decoding some packets. In the case of PLC (data==NULL)
* or FEC (decode_fec=1), then frame_size needs to be exactly
* the duration of audio that is missing, otherwise the
* decoder will not be in the optimal state to decode the
* next incoming packet. For the PLC and FEC cases, frame_size
* must be a multiple of 2.5 ms.
* @param decode_fec int: Flag (0 or 1) to request that any in-band
* forward error correction data be decoded.
* If no such data is available, the frame is
* decoded as if it were lost.
* @returns Number of samples decoded on success or a negative error code
* (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_multistream_decode(
OpusMSDecoder *st,
const unsigned char *data,
opus_int32 len,
opus_int16 *pcm,
int frame_size,
int decode_fec
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Decode a multistream Opus packet with floating point output.
* @param st OpusMSDecoder*: Multistream decoder state.
* @param[in] data const unsigned char*: Input payload.
* Use a NULL
* pointer to indicate packet
* loss.
* @param len opus_int32: Number of bytes in payload.
* @param[out] pcm opus_int16*: Output signal, with interleaved
* samples.
* This must contain room for
* frame_size*channels
* samples.
* @param frame_size int: The number of samples per channel of
* available space in \a pcm.
* If this is less than the maximum packet duration
* (120 ms; 5760 for 48kHz), this function will not be capable
* of decoding some packets. In the case of PLC (data==NULL)
* or FEC (decode_fec=1), then frame_size needs to be exactly
* the duration of audio that is missing, otherwise the
* decoder will not be in the optimal state to decode the
* next incoming packet. For the PLC and FEC cases, frame_size
* must be a multiple of 2.5 ms.
* @param decode_fec int: Flag (0 or 1) to request that any in-band
* forward error correction data be decoded.
* If no such data is available, the frame is
* decoded as if it were lost.
* @returns Number of samples decoded on success or a negative error code
* (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_multistream_decode_float(
OpusMSDecoder *st,
const unsigned char *data,
opus_int32 len,
float *pcm,
int frame_size,
int decode_fec
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Perform a CTL function on a multistream Opus decoder.
*
* Generally the request and subsequent arguments are generated by a
* convenience macro.
* @param st OpusMSDecoder*: Multistream decoder state.
* @param request This and all remaining parameters should be replaced by one
* of the convenience macros in @ref opus_genericctls,
* @ref opus_decoderctls, or @ref opus_multistream_ctls.
* @see opus_genericctls
* @see opus_decoderctls
* @see opus_multistream_ctls
*/
OPUS_EXPORT int opus_multistream_decoder_ctl(OpusMSDecoder *st, int request, ...) OPUS_ARG_NONNULL(1);
/** Frees an OpusMSDecoder allocated by
* opus_multistream_decoder_create().
* @param st OpusMSDecoder: Multistream decoder state to be freed.
*/
OPUS_EXPORT void opus_multistream_decoder_destroy(OpusMSDecoder *st);
/**@}*/
/**@}*/
#ifdef __cplusplus
}
#endif
#endif /* OPUS_MULTISTREAM_H */
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libopus/opus_projection.h
================================================
/* Copyright (c) 2017 Google Inc.
Written by Andrew Allen */
/*
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/**
* @file opus_projection.h
* @brief Opus projection reference API
*/
#ifndef OPUS_PROJECTION_H
#define OPUS_PROJECTION_H
#include "opus_multistream.h"
#ifdef __cplusplus
extern "C" {
#endif
/** @cond OPUS_INTERNAL_DOC */
/** These are the actual encoder and decoder CTL ID numbers.
* They should not be used directly by applications.c
* In general, SETs should be even and GETs should be odd.*/
/**@{*/
#define OPUS_PROJECTION_GET_DEMIXING_MATRIX_GAIN_REQUEST 6001
#define OPUS_PROJECTION_GET_DEMIXING_MATRIX_SIZE_REQUEST 6003
#define OPUS_PROJECTION_GET_DEMIXING_MATRIX_REQUEST 6005
/**@}*/
/** @endcond */
/** @defgroup opus_projection_ctls Projection specific encoder and decoder CTLs
*
* These are convenience macros that are specific to the
* opus_projection_encoder_ctl() and opus_projection_decoder_ctl()
* interface.
* The CTLs from @ref opus_genericctls, @ref opus_encoderctls,
* @ref opus_decoderctls, and @ref opus_multistream_ctls may be applied to a
* projection encoder or decoder as well.
*/
/**@{*/
/** Gets the gain (in dB. S7.8-format) of the demixing matrix from the encoder.
* @param[out] x opus_int32 *: Returns the gain (in dB. S7.8-format)
* of the demixing matrix.
* @hideinitializer
*/
#define OPUS_PROJECTION_GET_DEMIXING_MATRIX_GAIN(x) OPUS_PROJECTION_GET_DEMIXING_MATRIX_GAIN_REQUEST, __opus_check_int_ptr(x)
/** Gets the size in bytes of the demixing matrix from the encoder.
* @param[out] x opus_int32 *: Returns the size in bytes of the
* demixing matrix.
* @hideinitializer
*/
#define OPUS_PROJECTION_GET_DEMIXING_MATRIX_SIZE(x) OPUS_PROJECTION_GET_DEMIXING_MATRIX_SIZE_REQUEST, __opus_check_int_ptr(x)
/** Copies the demixing matrix to the supplied pointer location.
* @param[out] x unsigned char *: Returns the demixing matrix to the
* supplied pointer location.
* @param y opus_int32: The size in bytes of the reserved memory at the
* pointer location.
* @hideinitializer
*/
#define OPUS_PROJECTION_GET_DEMIXING_MATRIX(x,y) OPUS_PROJECTION_GET_DEMIXING_MATRIX_REQUEST, x, __opus_check_int(y)
/**@}*/
/** Opus projection encoder state.
* This contains the complete state of a projection Opus encoder.
* It is position independent and can be freely copied.
* @see opus_projection_ambisonics_encoder_create
*/
typedef struct OpusProjectionEncoder OpusProjectionEncoder;
/** Opus projection decoder state.
* This contains the complete state of a projection Opus decoder.
* It is position independent and can be freely copied.
* @see opus_projection_decoder_create
* @see opus_projection_decoder_init
*/
typedef struct OpusProjectionDecoder OpusProjectionDecoder;
/**\name Projection encoder functions */
/**@{*/
/** Gets the size of an OpusProjectionEncoder structure.
* @param channels int: The total number of input channels to encode.
* This must be no more than 255.
* @param mapping_family int: The mapping family to use for selecting
* the appropriate projection.
* @returns The size in bytes on success, or a negative error code
* (see @ref opus_errorcodes) on error.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_projection_ambisonics_encoder_get_size(
int channels,
int mapping_family
);
/** Allocates and initializes a projection encoder state.
* Call opus_projection_encoder_destroy() to release
* this object when finished.
* @param Fs opus_int32: Sampling rate of the input signal (in Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param channels int: Number of channels in the input signal.
* This must be at most 255.
* It may be greater than the number of
* coded channels (streams +
* coupled_streams).
* @param mapping_family int: The mapping family to use for selecting
* the appropriate projection.
* @param[out] streams int *: The total number of streams that will
* be encoded from the input.
* @param[out] coupled_streams int *: Number of coupled (2 channel)
* streams that will be encoded from the input.
* @param application int: The target encoder application.
* This must be one of the following:
*
*
#OPUS_APPLICATION_VOIP
*
Process signal for improved speech intelligibility.
*
#OPUS_APPLICATION_AUDIO
*
Favor faithfulness to the original input.
*
#OPUS_APPLICATION_RESTRICTED_LOWDELAY
*
Configure the minimum possible coding delay by disabling certain modes
* of operation.
*
* @param[out] error int *: Returns #OPUS_OK on success, or an error
* code (see @ref opus_errorcodes) on
* failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT OpusProjectionEncoder *opus_projection_ambisonics_encoder_create(
opus_int32 Fs,
int channels,
int mapping_family,
int *streams,
int *coupled_streams,
int application,
int *error
) OPUS_ARG_NONNULL(4) OPUS_ARG_NONNULL(5);
/** Initialize a previously allocated projection encoder state.
* The memory pointed to by \a st must be at least the size returned by
* opus_projection_ambisonics_encoder_get_size().
* This is intended for applications which use their own allocator instead of
* malloc.
* To reset a previously initialized state, use the #OPUS_RESET_STATE CTL.
* @see opus_projection_ambisonics_encoder_create
* @see opus_projection_ambisonics_encoder_get_size
* @param st OpusProjectionEncoder*: Projection encoder state to initialize.
* @param Fs opus_int32: Sampling rate of the input signal (in Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param channels int: Number of channels in the input signal.
* This must be at most 255.
* It may be greater than the number of
* coded channels (streams +
* coupled_streams).
* @param streams int: The total number of streams to encode from the
* input.
* This must be no more than the number of channels.
* @param coupled_streams int: Number of coupled (2 channel) streams
* to encode.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* encoded channels (streams +
* coupled_streams) must be no
* more than the number of input channels.
* @param application int: The target encoder application.
* This must be one of the following:
*
*
#OPUS_APPLICATION_VOIP
*
Process signal for improved speech intelligibility.
*
#OPUS_APPLICATION_AUDIO
*
Favor faithfulness to the original input.
*
#OPUS_APPLICATION_RESTRICTED_LOWDELAY
*
Configure the minimum possible coding delay by disabling certain modes
* of operation.
*
* @returns #OPUS_OK on success, or an error code (see @ref opus_errorcodes)
* on failure.
*/
OPUS_EXPORT int opus_projection_ambisonics_encoder_init(
OpusProjectionEncoder *st,
opus_int32 Fs,
int channels,
int mapping_family,
int *streams,
int *coupled_streams,
int application
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(5) OPUS_ARG_NONNULL(6);
/** Encodes a projection Opus frame.
* @param st OpusProjectionEncoder*: Projection encoder state.
* @param[in] pcm const opus_int16*: The input signal as interleaved
* samples.
* This must contain
* frame_size*channels
* samples.
* @param frame_size int: Number of samples per channel in the input
* signal.
* This must be an Opus frame size for the
* encoder's sampling rate.
* For example, at 48 kHz the permitted values
* are 120, 240, 480, 960, 1920, and 2880.
* Passing in a duration of less than 10 ms
* (480 samples at 48 kHz) will prevent the
* encoder from using the LPC or hybrid modes.
* @param[out] data unsigned char*: Output payload.
* This must contain storage for at
* least \a max_data_bytes.
* @param [in] max_data_bytes opus_int32: Size of the allocated
* memory for the output
* payload. This may be
* used to impose an upper limit on
* the instant bitrate, but should
* not be used as the only bitrate
* control. Use #OPUS_SET_BITRATE to
* control the bitrate.
* @returns The length of the encoded packet (in bytes) on success or a
* negative error code (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_projection_encode(
OpusProjectionEncoder *st,
const opus_int16 *pcm,
int frame_size,
unsigned char *data,
opus_int32 max_data_bytes
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2) OPUS_ARG_NONNULL(4);
/** Encodes a projection Opus frame from floating point input.
* @param st OpusProjectionEncoder*: Projection encoder state.
* @param[in] pcm const float*: The input signal as interleaved
* samples with a normal range of
* +/-1.0.
* Samples with a range beyond +/-1.0
* are supported but will be clipped by
* decoders using the integer API and
* should only be used if it is known
* that the far end supports extended
* dynamic range.
* This must contain
* frame_size*channels
* samples.
* @param frame_size int: Number of samples per channel in the input
* signal.
* This must be an Opus frame size for the
* encoder's sampling rate.
* For example, at 48 kHz the permitted values
* are 120, 240, 480, 960, 1920, and 2880.
* Passing in a duration of less than 10 ms
* (480 samples at 48 kHz) will prevent the
* encoder from using the LPC or hybrid modes.
* @param[out] data unsigned char*: Output payload.
* This must contain storage for at
* least \a max_data_bytes.
* @param [in] max_data_bytes opus_int32: Size of the allocated
* memory for the output
* payload. This may be
* used to impose an upper limit on
* the instant bitrate, but should
* not be used as the only bitrate
* control. Use #OPUS_SET_BITRATE to
* control the bitrate.
* @returns The length of the encoded packet (in bytes) on success or a
* negative error code (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_projection_encode_float(
OpusProjectionEncoder *st,
const float *pcm,
int frame_size,
unsigned char *data,
opus_int32 max_data_bytes
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(2) OPUS_ARG_NONNULL(4);
/** Frees an OpusProjectionEncoder allocated by
* opus_projection_ambisonics_encoder_create().
* @param st OpusProjectionEncoder*: Projection encoder state to be freed.
*/
OPUS_EXPORT void opus_projection_encoder_destroy(OpusProjectionEncoder *st);
/** Perform a CTL function on a projection Opus encoder.
*
* Generally the request and subsequent arguments are generated by a
* convenience macro.
* @param st OpusProjectionEncoder*: Projection encoder state.
* @param request This and all remaining parameters should be replaced by one
* of the convenience macros in @ref opus_genericctls,
* @ref opus_encoderctls, @ref opus_multistream_ctls, or
* @ref opus_projection_ctls
* @see opus_genericctls
* @see opus_encoderctls
* @see opus_multistream_ctls
* @see opus_projection_ctls
*/
OPUS_EXPORT int opus_projection_encoder_ctl(OpusProjectionEncoder *st, int request, ...) OPUS_ARG_NONNULL(1);
/**@}*/
/**\name Projection decoder functions */
/**@{*/
/** Gets the size of an OpusProjectionDecoder structure.
* @param channels int: The total number of output channels.
* This must be no more than 255.
* @param streams int: The total number of streams coded in the
* input.
* This must be no more than 255.
* @param coupled_streams int: Number streams to decode as coupled
* (2 channel) streams.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* coded channels (streams +
* coupled_streams) must be no
* more than 255.
* @returns The size in bytes on success, or a negative error code
* (see @ref opus_errorcodes) on error.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT opus_int32 opus_projection_decoder_get_size(
int channels,
int streams,
int coupled_streams
);
/** Allocates and initializes a projection decoder state.
* Call opus_projection_decoder_destroy() to release
* this object when finished.
* @param Fs opus_int32: Sampling rate to decode at (in Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param channels int: Number of channels to output.
* This must be at most 255.
* It may be different from the number of coded
* channels (streams +
* coupled_streams).
* @param streams int: The total number of streams coded in the
* input.
* This must be no more than 255.
* @param coupled_streams int: Number of streams to decode as coupled
* (2 channel) streams.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* coded channels (streams +
* coupled_streams) must be no
* more than 255.
* @param[in] demixing_matrix const unsigned char[demixing_matrix_size]: Demixing matrix
* that mapping from coded channels to output channels,
* as described in @ref opus_projection and
* @ref opus_projection_ctls.
* @param demixing_matrix_size opus_int32: The size in bytes of the
* demixing matrix, as
* described in @ref
* opus_projection_ctls.
* @param[out] error int *: Returns #OPUS_OK on success, or an error
* code (see @ref opus_errorcodes) on
* failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT OpusProjectionDecoder *opus_projection_decoder_create(
opus_int32 Fs,
int channels,
int streams,
int coupled_streams,
unsigned char *demixing_matrix,
opus_int32 demixing_matrix_size,
int *error
) OPUS_ARG_NONNULL(5);
/** Intialize a previously allocated projection decoder state object.
* The memory pointed to by \a st must be at least the size returned by
* opus_projection_decoder_get_size().
* This is intended for applications which use their own allocator instead of
* malloc.
* To reset a previously initialized state, use the #OPUS_RESET_STATE CTL.
* @see opus_projection_decoder_create
* @see opus_projection_deocder_get_size
* @param st OpusProjectionDecoder*: Projection encoder state to initialize.
* @param Fs opus_int32: Sampling rate to decode at (in Hz).
* This must be one of 8000, 12000, 16000,
* 24000, or 48000.
* @param channels int: Number of channels to output.
* This must be at most 255.
* It may be different from the number of coded
* channels (streams +
* coupled_streams).
* @param streams int: The total number of streams coded in the
* input.
* This must be no more than 255.
* @param coupled_streams int: Number of streams to decode as coupled
* (2 channel) streams.
* This must be no larger than the total
* number of streams.
* Additionally, The total number of
* coded channels (streams +
* coupled_streams) must be no
* more than 255.
* @param[in] demixing_matrix const unsigned char[demixing_matrix_size]: Demixing matrix
* that mapping from coded channels to output channels,
* as described in @ref opus_projection and
* @ref opus_projection_ctls.
* @param demixing_matrix_size opus_int32: The size in bytes of the
* demixing matrix, as
* described in @ref
* opus_projection_ctls.
* @returns #OPUS_OK on success, or an error code (see @ref opus_errorcodes)
* on failure.
*/
OPUS_EXPORT int opus_projection_decoder_init(
OpusProjectionDecoder *st,
opus_int32 Fs,
int channels,
int streams,
int coupled_streams,
unsigned char *demixing_matrix,
opus_int32 demixing_matrix_size
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(6);
/** Decode a projection Opus packet.
* @param st OpusProjectionDecoder*: Projection decoder state.
* @param[in] data const unsigned char*: Input payload.
* Use a NULL
* pointer to indicate packet
* loss.
* @param len opus_int32: Number of bytes in payload.
* @param[out] pcm opus_int16*: Output signal, with interleaved
* samples.
* This must contain room for
* frame_size*channels
* samples.
* @param frame_size int: The number of samples per channel of
* available space in \a pcm.
* If this is less than the maximum packet duration
* (120 ms; 5760 for 48kHz), this function will not be capable
* of decoding some packets. In the case of PLC (data==NULL)
* or FEC (decode_fec=1), then frame_size needs to be exactly
* the duration of audio that is missing, otherwise the
* decoder will not be in the optimal state to decode the
* next incoming packet. For the PLC and FEC cases, frame_size
* must be a multiple of 2.5 ms.
* @param decode_fec int: Flag (0 or 1) to request that any in-band
* forward error correction data be decoded.
* If no such data is available, the frame is
* decoded as if it were lost.
* @returns Number of samples decoded on success or a negative error code
* (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_projection_decode(
OpusProjectionDecoder *st,
const unsigned char *data,
opus_int32 len,
opus_int16 *pcm,
int frame_size,
int decode_fec
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Decode a projection Opus packet with floating point output.
* @param st OpusProjectionDecoder*: Projection decoder state.
* @param[in] data const unsigned char*: Input payload.
* Use a NULL
* pointer to indicate packet
* loss.
* @param len opus_int32: Number of bytes in payload.
* @param[out] pcm opus_int16*: Output signal, with interleaved
* samples.
* This must contain room for
* frame_size*channels
* samples.
* @param frame_size int: The number of samples per channel of
* available space in \a pcm.
* If this is less than the maximum packet duration
* (120 ms; 5760 for 48kHz), this function will not be capable
* of decoding some packets. In the case of PLC (data==NULL)
* or FEC (decode_fec=1), then frame_size needs to be exactly
* the duration of audio that is missing, otherwise the
* decoder will not be in the optimal state to decode the
* next incoming packet. For the PLC and FEC cases, frame_size
* must be a multiple of 2.5 ms.
* @param decode_fec int: Flag (0 or 1) to request that any in-band
* forward error correction data be decoded.
* If no such data is available, the frame is
* decoded as if it were lost.
* @returns Number of samples decoded on success or a negative error code
* (see @ref opus_errorcodes) on failure.
*/
OPUS_EXPORT OPUS_WARN_UNUSED_RESULT int opus_projection_decode_float(
OpusProjectionDecoder *st,
const unsigned char *data,
opus_int32 len,
float *pcm,
int frame_size,
int decode_fec
) OPUS_ARG_NONNULL(1) OPUS_ARG_NONNULL(4);
/** Perform a CTL function on a projection Opus decoder.
*
* Generally the request and subsequent arguments are generated by a
* convenience macro.
* @param st OpusProjectionDecoder*: Projection decoder state.
* @param request This and all remaining parameters should be replaced by one
* of the convenience macros in @ref opus_genericctls,
* @ref opus_decoderctls, @ref opus_multistream_ctls, or
* @ref opus_projection_ctls.
* @see opus_genericctls
* @see opus_decoderctls
* @see opus_multistream_ctls
* @see opus_projection_ctls
*/
OPUS_EXPORT int opus_projection_decoder_ctl(OpusProjectionDecoder *st, int request, ...) OPUS_ARG_NONNULL(1);
/** Frees an OpusProjectionDecoder allocated by
* opus_projection_decoder_create().
* @param st OpusProjectionDecoder: Projection decoder state to be freed.
*/
OPUS_EXPORT void opus_projection_decoder_destroy(OpusProjectionDecoder *st);
/**@}*/
/**@}*/
#ifdef __cplusplus
}
#endif
#endif /* OPUS_PROJECTION_H */
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/libopus/opus_types.h
================================================
/* (C) COPYRIGHT 1994-2002 Xiph.Org Foundation */
/* Modified by Jean-Marc Valin */
/*
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/* opus_types.h based on ogg_types.h from libogg */
/**
@file opus_types.h
@brief Opus reference implementation types
*/
#ifndef OPUS_TYPES_H
#define OPUS_TYPES_H
#define opus_int int /* used for counters etc; at least 16 bits */
#define opus_int64 long long
#define opus_int8 signed char
#define opus_uint unsigned int /* used for counters etc; at least 16 bits */
#define opus_uint64 unsigned long long
#define opus_uint8 unsigned char
/* Use the real stdint.h if it's there (taken from Paul Hsieh's pstdint.h) */
#if (defined(__STDC__) && __STDC__ && defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L) || (defined(__GNUC__) && (defined(_STDINT_H) || defined(_STDINT_H_)) || defined (HAVE_STDINT_H))
#include
# undef opus_int64
# undef opus_int8
# undef opus_uint64
# undef opus_uint8
typedef int8_t opus_int8;
typedef uint8_t opus_uint8;
typedef int16_t opus_int16;
typedef uint16_t opus_uint16;
typedef int32_t opus_int32;
typedef uint32_t opus_uint32;
typedef int64_t opus_int64;
typedef uint64_t opus_uint64;
#elif defined(_WIN32)
# if defined(__CYGWIN__)
# include <_G_config.h>
typedef _G_int32_t opus_int32;
typedef _G_uint32_t opus_uint32;
typedef _G_int16 opus_int16;
typedef _G_uint16 opus_uint16;
# elif defined(__MINGW32__)
typedef short opus_int16;
typedef unsigned short opus_uint16;
typedef int opus_int32;
typedef unsigned int opus_uint32;
# elif defined(__MWERKS__)
typedef int opus_int32;
typedef unsigned int opus_uint32;
typedef short opus_int16;
typedef unsigned short opus_uint16;
# else
/* MSVC/Borland */
typedef __int32 opus_int32;
typedef unsigned __int32 opus_uint32;
typedef __int16 opus_int16;
typedef unsigned __int16 opus_uint16;
# endif
#elif defined(__MACOS__)
# include
typedef SInt16 opus_int16;
typedef UInt16 opus_uint16;
typedef SInt32 opus_int32;
typedef UInt32 opus_uint32;
#elif (defined(__APPLE__) && defined(__MACH__)) /* MacOS X Framework build */
# include
typedef int16_t opus_int16;
typedef u_int16_t opus_uint16;
typedef int32_t opus_int32;
typedef u_int32_t opus_uint32;
#elif defined(__BEOS__)
/* Be */
# include
typedef int16 opus_int16;
typedef u_int16 opus_uint16;
typedef int32_t opus_int32;
typedef u_int32_t opus_uint32;
#elif defined (__EMX__)
/* OS/2 GCC */
typedef short opus_int16;
typedef unsigned short opus_uint16;
typedef int opus_int32;
typedef unsigned int opus_uint32;
#elif defined (DJGPP)
/* DJGPP */
typedef short opus_int16;
typedef unsigned short opus_uint16;
typedef int opus_int32;
typedef unsigned int opus_uint32;
#elif defined(R5900)
/* PS2 EE */
typedef int opus_int32;
typedef unsigned opus_uint32;
typedef short opus_int16;
typedef unsigned short opus_uint16;
#elif defined(__SYMBIAN32__)
/* Symbian GCC */
typedef signed short opus_int16;
typedef unsigned short opus_uint16;
typedef signed int opus_int32;
typedef unsigned int opus_uint32;
#elif defined(CONFIG_TI_C54X) || defined (CONFIG_TI_C55X)
typedef short opus_int16;
typedef unsigned short opus_uint16;
typedef long opus_int32;
typedef unsigned long opus_uint32;
#elif defined(CONFIG_TI_C6X)
typedef short opus_int16;
typedef unsigned short opus_uint16;
typedef int opus_int32;
typedef unsigned int opus_uint32;
#else
/* Give up, take a reasonable guess */
typedef short opus_int16;
typedef unsigned short opus_uint16;
typedef int opus_int32;
typedef unsigned int opus_uint32;
#endif
#endif /* OPUS_TYPES_H */
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/ogg_opus_encoder.cc
================================================
/*
* Copyright 2019 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "ogg_opus_encoder.h"
#include
#include
#include
#include
#include
#include "opus_tools/opus_header.h"
// Ogg Opus information comes from the standard here:
// https://tools.ietf.org/html/draft-ietf-codec-oggopus-14
// This was also a useful reference:
// https://github.com/krad-radio/butt-krad-opus/blob/302a4b6a6a596be6632f30cfb567a6a6d8fcb3f9/src/opus_encode.cpp
namespace audio_util {
namespace {
static constexpr float kEncodingBufferSizeSeconds = 0.02f;
std::string SerializeUint32(uint32_t value) {
std::string result(4, '\0');
#if __BYTE_ORDER == __BIG_ENDIAN
#error The following line assumes that the byte order is little endian.
#endif
memcpy(&result[0], &value, sizeof(value));
return result;
}
} // namespace
OggOpusEncoder::OggOpusEncoder(int num_channels, int sample_rate_hz,
int bitrate_bps, bool use_vbr,
bool low_latency_mode)
: num_channels_(num_channels),
sample_rate_hz_(sample_rate_hz),
frame_size_(kEncodingBufferSizeSeconds * sample_rate_hz_),
encoder_(OpusUniquePtr(
opus_encoder_create(sample_rate_hz_, num_channels_,
OPUS_APPLICATION_AUDIO, &error_code_),
opus_encoder_destroy)),
flushed_(false),
low_latency_mode_(low_latency_mode),
elements_in_pcm_frame_(0),
pcm_frame_(num_channels_ * frame_size_) {
assert(num_channels <= 2); // Only mono and stereo are supported).
std::vector valid_sample_rates = {8000, 12000, 16000, 24000, 48000};
assert(std::find(valid_sample_rates.begin(), valid_sample_rates.end(),
sample_rate_hz) != valid_sample_rates.end());
assert(bitrate_bps >= 500);
assert(bitrate_bps <= 512000);
opus_encoder_ctl(encoder_.get(), OPUS_SET_BITRATE(bitrate_bps));
if (!use_vbr) {
opus_encoder_ctl(encoder_.get(), OPUS_SET_VBR(0));
}
constexpr int kComplexity = 4;
opus_encoder_ctl(encoder_.get(), OPUS_SET_COMPLEXITY(kComplexity));
// We will always pass exactly one frame at a time to the encoder.
opus_frame_.resize(kBytesPerSample * pcm_frame_.size());
// Start generating Ogg packets (though they don't get sent out until the
// first call to Encode()).
packet_count_ = 0;
granule_position_ = 0;
ogg_stream_init(&stream_, 0 /* serial number */);
GenerateOggPacketsForHeader();
}
OggOpusEncoder::~OggOpusEncoder() {
ogg_stream_clear(&stream_);
}
// Encodes 16-bit PCM data in OggOpus format.
const std::vector& OggOpusEncoder::Process(
const std::vector& pcm) {
assert(!flushed_);
assert((pcm.size() % num_channels_) == 0);
if (!header_) {
ogg_bytes_.resize(0);
} else {
header_ = false;
}
int num_samples_processed = 0;
// Process the first block, handling any leftovers from a previous round.
if (elements_in_pcm_frame_ > 0) {
int entries_to_write =
std::min(pcm_frame_.size() - elements_in_pcm_frame_, pcm.size());
std::copy(pcm.begin(), pcm.begin() + entries_to_write,
pcm_frame_.begin() + elements_in_pcm_frame_);
elements_in_pcm_frame_ += entries_to_write;
if (elements_in_pcm_frame_ == pcm_frame_.size()) {
// pcm_frame_ is full, encode it.
int num_opus_frame_bytes =
opus_encode(encoder_.get(), pcm_frame_.data(), frame_size_,
opus_frame_.data(), opus_frame_.size());
assert(num_opus_frame_bytes >= 0);
GenerateOggPacketsForOpusFrame(opus_frame_.data(), num_opus_frame_bytes,
&ogg_bytes_, false);
num_samples_processed += entries_to_write;
pcm_frame_.assign(pcm_frame_.size(), 0);
elements_in_pcm_frame_ = 0;
} else {
// There's nothing to encode. We've put all of pcm data into pcm_frame_
// for later processing.
ogg_bytes_.resize(0);
return ogg_bytes_;
}
}
// Process whole frames directly from pcm.
while (num_samples_processed + frame_size_ * num_channels_ <= pcm.size()) {
int num_opus_frame_bytes =
opus_encode(encoder_.get(), pcm.data() + num_samples_processed,
frame_size_, opus_frame_.data(), opus_frame_.size());
assert(num_opus_frame_bytes >= 0);
GenerateOggPacketsForOpusFrame(opus_frame_.data(), num_opus_frame_bytes,
&ogg_bytes_, false);
num_samples_processed += frame_size_ * num_channels_;
}
if (low_latency_mode_) {
// Force the codec to produce samples for every input buffer.
AppendOggStateToBuffer(&ogg_bytes_, true);
}
// Place any remaining samples in pcm_frame_.
elements_in_pcm_frame_ = pcm.size() - num_samples_processed;
std::copy(pcm.begin() + num_samples_processed, pcm.end(), pcm_frame_.begin());
return ogg_bytes_;
}
// Returns any remaining samples from the codec.
const std::vector& OggOpusEncoder::Flush() {
assert(!flushed_);
ogg_bytes_.resize(0);
int num_opus_frame_bytes =
opus_encode(encoder_.get(), pcm_frame_.data(), frame_size_,
opus_frame_.data(), opus_frame_.size());
assert(num_opus_frame_bytes >= 0);
GenerateOggPacketsForOpusFrame(opus_frame_.data(), num_opus_frame_bytes,
&ogg_bytes_, true);
flushed_ = true;
return ogg_bytes_;
}
void OggOpusEncoder::GenerateOggPacketsForHeader() {
// Both header packets must have granule position of zero.
assert(granule_position_ == 0);
OpusHeader header;
header.version = 1;
header.channels = num_channels_;
opus_encoder_ctl(encoder_.get(), OPUS_GET_LOOKAHEAD(&header.preskip));
header.input_sample_rate = sample_rate_hz_;
header.gain = 0;
header.channel_mapping = 0;
// Write the ID header.
ogg_packet id_packet;
id_packet.b_o_s = 1; // The first packet.
id_packet.e_o_s = 0;
id_packet.granulepos = granule_position_;
id_packet.packetno = packet_count_;
constexpr int kHeaderSizeUpperBound = 64;
id_packet.packet = new unsigned char[kHeaderSizeUpperBound];
// opus_header_to_packet fills id_packet.packet with header data and returns
// the number of bytes.
id_packet.bytes =
opus_header_to_packet(&header, id_packet.packet, kHeaderSizeUpperBound);
// Add the ID packet into the stream.
packet_count_++;
ogg_stream_packetin(&stream_, &id_packet);
// Write the comment header.
ogg_packet comment_packet;
comment_packet.b_o_s = 0;
comment_packet.e_o_s = 0;
comment_packet.granulepos = granule_position_;
comment_packet.packetno = packet_count_;
const std::string kVendor = "Google using libopus";
std::string packet = "";
packet.append("OpusTags");
packet.append(SerializeUint32(kVendor.size()));
packet.append(kVendor);
packet.append(SerializeUint32(0));
comment_packet.packet = const_cast(
reinterpret_cast(packet.c_str()));
comment_packet.bytes = packet.length();
// Add the comment header into the stream.
packet_count_++;
ogg_stream_packetin(&stream_, &comment_packet);
// Force a page break after the comment header.
// According to
// https://tools.ietf.org/html/draft-ietf-codec-oggopus-14#section-3 there is
// a mandatory page break after the comment header.
AppendOggStateToBuffer(&ogg_bytes_, true);
header_ = true;
delete[] id_packet.packet;
}
void OggOpusEncoder::GenerateOggPacketsForOpusFrame(
unsigned char* opus_frame_bytes, int opus_bytes_length,
std::vector* ogg_bytes, bool flush) {
// Flush data from the ogg object into the outgoing stream.
AppendOggStateToBuffer(ogg_bytes, flush);
// Write the most recent buffer of Opus data into an Ogg packet.
ogg_packet frame_packet;
frame_packet.b_o_s = 0;
frame_packet.e_o_s = flush ? 1 : 0;
// According to
// https://tools.ietf.org/html/draft-ietf-codec-oggopus-14#section-4 the
// granule position should include all samples up to the last packet completed
// on the page, so we need to update granule_position_ before assigning it to
// the packet. If we're closing the stream, we don't assume that the last
// packet includes a full frame.
if (flush) {
granule_position_ += (elements_in_pcm_frame_ / num_channels_);
} else {
granule_position_ += frame_size_;
}
frame_packet.granulepos = granule_position_;
frame_packet.packetno = packet_count_;
frame_packet.packet = opus_frame_bytes;
frame_packet.bytes = opus_bytes_length;
// Add the data packet into the stream.
packet_count_++;
ogg_stream_packetin(&stream_, &frame_packet);
// Try flushing again after data packet.
AppendOggStateToBuffer(ogg_bytes, flush);
}
void OggOpusEncoder::AppendOggStateToBuffer(std::vector* buffer,
bool flush_ogg_stream) {
int (*write_fun)(ogg_stream_state*, ogg_page*) =
flush_ogg_stream ? &ogg_stream_flush : &ogg_stream_pageout;
while (write_fun(&stream_, &page_) != 0) {
const int initial_size = buffer->size();
buffer->resize(buffer->size() + page_.header_len + page_.body_len);
memcpy(buffer->data() + initial_size, page_.header, page_.header_len);
memcpy(buffer->data() + initial_size + page_.header_len, page_.body,
page_.body_len);
}
}
} // namespace audio_util
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/ogg_opus_encoder.h
================================================
/*
* Copyright 2019 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef AUDIO_UTIL_OGG_OPUS_ENCODER_H_
#define AUDIO_UTIL_OGG_OPUS_ENCODER_H_
#include
#include
#include
#include
#include "libogg/ogg.h"
#include "libopus/opus.h"
namespace audio_util {
// This class is meant to be a dependency-light streaming encoder.
//
// Encoding is done internally on a block size of 20ms, which seems to be the
// recommended size for Opus encoding.
class OggOpusEncoder {
public:
// Input is int16 data.
constexpr static int kBytesPerSample = 2;
constexpr static int kBitsPerSample = 16;
// num_channels must be 1 or 2.
// sample rate must be one of {8000, 12000, 16000, 24000, 48000}
// Note that low_latency_mode will increase the total number of Ogg packets,
// but will reduce overall latency of the codec. This does not impact the
// quality of audio compression, only how the data is packaged in the Ogg
// container. Low latency mode is only recommended for realtime streaming
// applications. See test for actual bitrate increases.
OggOpusEncoder(int num_channels, int sample_rate_hz, int bitrate_bps,
bool use_vbr, bool low_latency_mode);
~OggOpusEncoder();
// Encodes 16-bit PCM data in OggOpus format. There is no restriction on the
// size of the input vector.
// Note that it is very common for the returned vector to be empty. Keep
// calling this function with new samples until they become available.
const std::vector& Process(const std::vector& pcm);
// Returns any remaining samples from the codec. This should be called last,
// and never more than once.
const std::vector& Flush();
private:
using OpusUniquePtr =
std::unique_ptr;
std::string GetOpusErrorMessage() const;
// Push the Opus header details into the ogg stream.
void GenerateOggPacketsForHeader();
// Push data from a single Opus frame into an Ogg stream.
void GenerateOggPacketsForOpusFrame(unsigned char* opus_frame_bytes,
int opus_bytes_length,
std::vector* ogg_bytes,
bool flush);
// Moves data from the stream_ object into buffer.
void AppendOggStateToBuffer(std::vector* buffer, bool flush);
int num_channels_;
int sample_rate_hz_;
int bitrate_bps_;
// Number of samples in an Opus 20ms frame for a single channel.
int frame_size_;
OpusUniquePtr encoder_;
// Stores the status of Opus codec initialization.
int error_code_;
// Checks that Flush() isn't called multiple times.
bool flushed_;
bool header_;
// When true, flushing of the Ogg stream after every call to Process().
bool low_latency_mode_;
// A preallocated buffer to store the temporary OGG result.
std::vector opus_frame_;
std::vector ogg_bytes_;
// A stored buffer for a single frame of PCM data to be processed.
int elements_in_pcm_frame_;
std::vector pcm_frame_;
// Ogg objects.
ogg_stream_state stream_;
ogg_page page_;
int packet_count_; // Count of packets pushed to the stream.
int granule_position_; // Position in the ogg stream.
};
} // namespace audio_util
#endif // AUDIO_UTIL_OGG_OPUS_ENCODER_H_
================================================
FILE: SGM_android/SmartGlassesManager/src/main/cpp/opus_tools/opus_header.h
================================================
/* Copyright (C)2012 Xiph.Org Foundation
File: opus_header.h
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef OPUS_HEADER_H
#define OPUS_HEADER_H
#include "../libogg/ogg.h"
#ifdef __cplusplus
extern "C" {
#endif
typedef struct {
int version;
int channels; /* Number of channels: 1..255 */
int preskip;
ogg_uint32_t input_sample_rate;
int gain; /* in dB S7.8 should be zero whenever possible */
int channel_mapping;
/* The rest is only used if channel_mapping != 0 */
int nb_streams;
int nb_coupled;
unsigned char stream_map[255];
} OpusHeader;
int opus_header_parse(const unsigned char *header, int len, OpusHeader *h);
int opus_header_to_packet(const OpusHeader *h, unsigned char *packet, int len);
extern const int wav_permute_matrix[8][8];
#ifdef __cplusplus
}
#endif
#endif
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/SmartGlassesAndroidService.java
================================================
package com.teamopensmartglasses.smartglassesmanager;
import android.app.Notification;
import android.app.NotificationChannel;
import android.app.NotificationManager;
import android.app.PendingIntent;
import android.app.Service;
import android.content.Context;
import android.content.DialogInterface;
import android.content.Intent;
import android.graphics.Bitmap;
import android.os.Binder;
import android.os.Bundle;
import android.os.Handler;
import android.os.IBinder;
import android.util.Log;
import android.widget.Toast;
import androidx.core.app.NotificationCompat;
import androidx.lifecycle.LifecycleService;
import androidx.preference.PreferenceManager;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.PostGenericGlobalMessageEvent;
import com.teamopensmartglasses.augmentoslib.events.SmartGlassesConnectedEvent;
import com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators.SmartGlassesFontSize;
import com.teamopensmartglasses.smartglassesmanager.comms.MessageTypes;
import com.teamopensmartglasses.augmentoslib.events.BulletPointListViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.CenteredTextViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.DoubleTextWallViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.FinalScrollingTextRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.HomeScreenEvent;
import com.teamopensmartglasses.augmentoslib.events.ReferenceCardImageViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.ReferenceCardSimpleViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.RowsCardViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.SendBitmapViewRequestEvent;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.SetFontSizeEvent;
import com.teamopensmartglasses.augmentoslib.events.TextWallViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.ScrollingTextViewStartRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.ScrollingTextViewStopRequestEvent;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.SmartGlassesConnectionEvent;
import com.teamopensmartglasses.augmentoslib.events.TextLineViewRequestEvent;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.TextToSpeechEvent;
import com.teamopensmartglasses.smartglassesmanager.speechrecognition.ASR_FRAMEWORKS;
import com.teamopensmartglasses.smartglassesmanager.speechrecognition.SpeechRecSwitchSystem;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.AudioWearable;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.InmoAirOne;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.SmartGlassesDevice;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.SmartGlassesOperatingSystem;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.TCLRayNeoXTwo;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.VuzixShield;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.VuzixUltralite;
import com.teamopensmartglasses.smartglassesmanager.texttospeech.TextToSpeechSystem;
import org.greenrobot.eventbus.EventBus;
import org.greenrobot.eventbus.EventBusException;
import org.greenrobot.eventbus.Subscribe;
import org.json.JSONObject;
import java.util.ArrayList;
import java.util.Arrays;
import io.reactivex.rxjava3.subjects.PublishSubject;
/** Main service of Smart Glasses Manager, that starts connections to smart glasses and talks to third party apps (3PAs) */
public abstract class SmartGlassesAndroidService extends LifecycleService {
private static final String TAG = "SGM_ASP_Service";
// Service Binder given to clients
private final IBinder binder = new LocalBinder();
public static final String ACTION_START_FOREGROUND_SERVICE = "MY_ACTION_START_FOREGROUND_SERVICE";
public static final String ACTION_STOP_FOREGROUND_SERVICE = "MY_ACTION_STOP_FOREGROUND_SERVICE";
private int myNotificationId;
private Class mainActivityClass;
private String myChannelId;
private String notificationAppName;
private String notificationDescription;
private int notificationDrawable;
//Text to Speech
private TextToSpeechSystem textToSpeechSystem;
//observables to send data around app
PublishSubject dataObservable;
//representatives of the other pieces of the system
SmartGlassesRepresentative smartGlassesRepresentative;
//speech rec
SpeechRecSwitchSystem speechRecSwitchSystem;
//connection handler
public Handler connectHandler;
public SmartGlassesAndroidService(Class mainActivityClass, String myChannelId, int myNotificationId, String notificationAppName, String notificationDescription, int notificationDrawable){
this.myNotificationId = myNotificationId;
this.mainActivityClass = mainActivityClass;
this.myChannelId = myChannelId;
this.notificationAppName = notificationAppName;
this.notificationDescription = notificationDescription;
this.notificationDrawable = notificationDrawable;
}
@Override
public void onCreate() {
super.onCreate();
//setup connection handler
connectHandler = new Handler();
//start speech rec
speechRecSwitchSystem = new SpeechRecSwitchSystem(this.getApplicationContext());
ASR_FRAMEWORKS asrFramework = getChosenAsrFramework(this.getApplicationContext());
String transcribeLanguage = getChosenTranscribeLanguage(this.getApplicationContext());
String targetLanguage = getChosenTargetLanguage(this.getApplicationContext());
String sourceLanguage = getChosenSourceLanguage(this.getApplicationContext());
int selectedLiveCaptionsTranslation = getSelectedLiveCaptionsTranslation(this.getApplicationContext());
if (selectedLiveCaptionsTranslation != 2) speechRecSwitchSystem.startAsrFramework(asrFramework, transcribeLanguage);
else {
if (transcribeLanguage.equals(sourceLanguage)) speechRecSwitchSystem.startAsrFramework(asrFramework, transcribeLanguage, targetLanguage); // If transcribe language and source language are the same translate to the target language
else speechRecSwitchSystem.startAsrFramework(asrFramework, transcribeLanguage, sourceLanguage);
}
// speechRecSwitchSystem.startAsrFramework(asrFramework, "Chinese (Hanzi)", "English");
//setup data observable which passes information (transcripts, commands, etc. around our app using mutlicasting
dataObservable = PublishSubject.create();
//start text to speech
textToSpeechSystem = new TextToSpeechSystem(this);
textToSpeechSystem.setup();
}
protected void setupEventBusSubscribers() {
try {
EventBus.getDefault().register(this);
}
catch(EventBusException e){
e.printStackTrace();
}
}
@Subscribe
public void handleConnectionEvent(SmartGlassesConnectionEvent event) {
sendUiUpdate();
}
protected abstract void onGlassesConnected(SmartGlassesDevice device);
public void connectToSmartGlasses(SmartGlassesDevice device) {
//this represents the smart glasses - it handles the connection, sending data to them, etc
LifecycleService currContext = this;
connectHandler.post(new Runnable() {
@Override
public void run() {
Log.d(TAG, "CONNECTING TO SMART GLASSES");
smartGlassesRepresentative = new SmartGlassesRepresentative(currContext, device, currContext, dataObservable);
smartGlassesRepresentative.connectToSmartGlasses();
}
});
}
@Override
public void onDestroy() {
Log.d(TAG, "WearableAiAspService killing itself and all its children");
EventBus.getDefault().unregister(this);
//kill speech rec
if (speechRecSwitchSystem != null){
speechRecSwitchSystem.destroy();
}
//kill asg connection
if (smartGlassesRepresentative != null) {
smartGlassesRepresentative.destroy();
smartGlassesRepresentative = null;
}
//kill data transmitters
if (dataObservable != null) {
dataObservable.onComplete();
}
//kill textToSpeech
textToSpeechSystem.destroy();
//kill aioConnect
aioRetryHandler.removeCallbacks(aioRetryConnectionTask);
//call parent destroy
super.onDestroy();
Log.d(TAG, "WearableAiAspService destroy complete");
}
public void sendTestCard(String title, String body, String img) {
Log.d(TAG, "SENDING TEST CARD FROM WAIService");
EventBus.getDefault().post(new ReferenceCardSimpleViewRequestEvent(title, body));
}
public int getSmartGlassesConnectState() {
if (smartGlassesRepresentative != null) {
return smartGlassesRepresentative.getConnectionState();
} else {
return 0;
}
}
public SmartGlassesDevice getConnectedSmartGlasses() {
if (smartGlassesRepresentative == null) return null;
if(smartGlassesRepresentative.getConnectionState() != 2) return null;
return smartGlassesRepresentative.smartGlassesDevice;
}
public SmartGlassesOperatingSystem getConnectedDeviceModelOs(){
if (smartGlassesRepresentative == null) return null;
if(smartGlassesRepresentative.getConnectionState() != 2) return null;
return smartGlassesRepresentative.smartGlassesDevice.glassesOs;
}
public void sendUiUpdate() {
//connectionState = 2 means connected
Intent intent = new Intent();
intent.setAction(MessageTypes.GLASSES_STATUS_UPDATE);
// Set the optional additional information in extra field.
int connectionState;
if (smartGlassesRepresentative != null) {
connectionState = smartGlassesRepresentative.getConnectionState();
intent.putExtra(MessageTypes.CONNECTION_GLASSES_GLASSES_OBJECT, smartGlassesRepresentative.smartGlassesDevice);
// Update preferred wearable if connected
if(connectionState == 2){
savePreferredWearable(this, smartGlassesRepresentative.smartGlassesDevice.deviceModelName);
onGlassesConnected(smartGlassesRepresentative.smartGlassesDevice);
EventBus.getDefault().post(new SmartGlassesConnectedEvent(smartGlassesRepresentative.smartGlassesDevice));
}
} else {
connectionState = 0;
}
intent.putExtra(MessageTypes.CONNECTION_GLASSES_STATUS_UPDATE, connectionState);
sendBroadcast(intent);
}
/** Saves the chosen ASR framework in user shared preference. */
public static void saveChosenAsrFramework(Context context, ASR_FRAMEWORKS asrFramework) {
PreferenceManager.getDefaultSharedPreferences(context)
.edit()
.putString(context.getResources().getString(R.string.SHARED_PREF_ASR_KEY), asrFramework.name())
.apply();
}
/** Gets the chosen ASR framework from shared preference. */
public static ASR_FRAMEWORKS getChosenAsrFramework(Context context) {
String asrString = PreferenceManager.getDefaultSharedPreferences(context).getString(context.getResources().getString(R.string.SHARED_PREF_ASR_KEY), "");
if (asrString.equals("")){
saveChosenAsrFramework(context, ASR_FRAMEWORKS.AZURE_ASR_FRAMEWORK);
asrString = ASR_FRAMEWORKS.AZURE_ASR_FRAMEWORK.name();
}
return ASR_FRAMEWORKS.valueOf(asrString);
}
public void changeChosenAsrFramework(ASR_FRAMEWORKS asrFramework){
saveChosenAsrFramework(getApplicationContext(), asrFramework);
if (speechRecSwitchSystem != null) {
speechRecSwitchSystem.startAsrFramework(asrFramework);
}
}
/** Gets the API key from shared preference. */
public static String getApiKey(Context context) {
return PreferenceManager.getDefaultSharedPreferences(context).getString(context.getResources().getString(R.string.SHARED_PREF_KEY), "");
}
/** Saves the API Key in user shared preference. */
public static void saveApiKey(Context context, String key) {
PreferenceManager.getDefaultSharedPreferences(context)
.edit()
.putString(context.getResources().getString(R.string.SHARED_PREF_KEY), key)
.apply();
}
/** Gets the preferred wearable from shared preference. */
public static String getPreferredWearable(Context context) {
return PreferenceManager.getDefaultSharedPreferences(context).getString(context.getResources().getString(R.string.PREFERRED_WEARABLE), "");
}
/** Saves the preferred wearable in user shared preference. */
public static void savePreferredWearable(Context context, String wearableName) {
PreferenceManager.getDefaultSharedPreferences(context)
.edit()
.putString(context.getResources().getString(R.string.PREFERRED_WEARABLE), wearableName)
.apply();
}
public static void saveChosenTranscribeLanguage(Context context, String transcribeLanguageString) {
PreferenceManager.getDefaultSharedPreferences(context)
.edit()
.putString(context.getResources().getString(R.string.SHARED_PREF_TRANSCRIBE_LANGUAGE), transcribeLanguageString)
.apply();
}
public static String getChosenTranscribeLanguage(Context context) {
String transcribeLanguageString = PreferenceManager.getDefaultSharedPreferences(context).getString(context.getResources().getString(R.string.SHARED_PREF_TRANSCRIBE_LANGUAGE), "");
if (transcribeLanguageString.equals("")){
saveChosenTranscribeLanguage(context, "English");
transcribeLanguageString = "English";
}
return transcribeLanguageString;
}
public static void saveChosenTargetLanguage(Context context, String targetLanguageString) {
PreferenceManager.getDefaultSharedPreferences(context)
.edit()
.putString(context.getResources().getString(R.string.SHARED_PREF_TARGET_LANGUAGE), targetLanguageString)
.apply();
}
public static String getChosenTargetLanguage(Context context) {
String targetLanguageString = PreferenceManager.getDefaultSharedPreferences(context).getString(context.getResources().getString(R.string.SHARED_PREF_TARGET_LANGUAGE), "");
if (targetLanguageString.equals("")){
saveChosenTargetLanguage(context, "English");
targetLanguageString = "English";
}
return targetLanguageString;
}
public static void saveChosenSourceLanguage(Context context, String sourceLanguageString) {
PreferenceManager.getDefaultSharedPreferences(context)
.edit()
.putString(context.getResources().getString(R.string.SHARED_PREF_SOURCE_LANGUAGE), sourceLanguageString)
.apply();
}
public static String getChosenSourceLanguage(Context context) {
String sourceLanguageString = PreferenceManager.getDefaultSharedPreferences(context).getString(context.getResources().getString(R.string.SHARED_PREF_SOURCE_LANGUAGE), "");
if (sourceLanguageString.equals("")){
saveChosenSourceLanguage(context, "English");
sourceLanguageString = "English";
}
return sourceLanguageString;
}
public static void saveSelectedLiveCaptionsTranslationChecked(Context context, int liveCaptionsTranslationSelected) {
PreferenceManager.getDefaultSharedPreferences(context)
.edit()
.putInt(context.getResources().getString(R.string.SHARED_PREF_LIVE_CAPTIONS_TRANSLATION), liveCaptionsTranslationSelected)
.apply();
}
public static int getSelectedLiveCaptionsTranslation(Context context) {
return PreferenceManager.getDefaultSharedPreferences(context).getInt(context.getResources().getString(R.string.SHARED_PREF_LIVE_CAPTIONS_TRANSLATION), 0);
}
//switches the currently running transcribe language without changing the default/saved language
public void switchRunningTranscribeLanguage(String language){
if (speechRecSwitchSystem.currentLanguage.equals(language)){
return;
}
//kill previous speech rec
speechRecSwitchSystem.destroy();
speechRecSwitchSystem = null;
//start speech rec after small delay
Handler speechRecHandler = new Handler();
Context context = this;
speechRecHandler.postDelayed(new Runnable() {
@Override
public void run() {
speechRecSwitchSystem = new SpeechRecSwitchSystem(context);
ASR_FRAMEWORKS asrFramework = getChosenAsrFramework(context);
speechRecSwitchSystem.startAsrFramework(asrFramework, language);
}
}, 250);
}
//service stuff
private Notification updateNotification() {
Context context = getApplicationContext();
PendingIntent action = PendingIntent.getActivity(context,
0, new Intent(context, mainActivityClass),
PendingIntent.FLAG_CANCEL_CURRENT | PendingIntent.FLAG_MUTABLE); // Flag indicating that if the described PendingIntent already exists, the current one should be canceled before generating a new one.
NotificationManager manager = (NotificationManager) getSystemService(Context.NOTIFICATION_SERVICE);
NotificationCompat.Builder builder;
String CHANNEL_ID = myChannelId;
NotificationChannel channel = new NotificationChannel(CHANNEL_ID, notificationAppName,
NotificationManager.IMPORTANCE_HIGH);
channel.setDescription(notificationDescription);
manager.createNotificationChannel(channel);
builder = new NotificationCompat.Builder(this, CHANNEL_ID);
return builder.setContentIntent(action)
.setContentTitle(notificationAppName)
.setContentText(notificationDescription)
.setSmallIcon(notificationDrawable)
.setTicker("...")
.setContentIntent(action)
.setOngoing(true).build();
}
public class LocalBinder extends Binder {
public SmartGlassesAndroidService getService() {
// Return this instance of LocalService so clients can call public methods
return SmartGlassesAndroidService.this;
}
}
@Override
public IBinder onBind(Intent intent) {
super.onBind(intent);
return binder;
}
@Override
public int onStartCommand(Intent intent, int flags, int startId) {
super.onStartCommand(intent, flags, startId);
if (intent != null) {
String action = intent.getAction();
Bundle extras = intent.getExtras();
switch (action) {
case ACTION_START_FOREGROUND_SERVICE:
// start the service in the foreground
Log.d("TEST", "starting foreground");
startForeground(myNotificationId, updateNotification());
break;
case ACTION_STOP_FOREGROUND_SERVICE:
stopForeground(true);
stopSelf();
break;
}
}
return Service.START_STICKY;
}
// Setup for aioConnectSmartGlasses
ArrayList smartGlassesDevices = new ArrayList<>();
Handler aioRetryHandler = new Handler();
Runnable aioRetryConnectionTask = new Runnable() {
@Override
public void run() {
if (smartGlassesRepresentative == null || smartGlassesRepresentative.getConnectionState() != 2) { // If still disconnected
if(!smartGlassesDevices.isEmpty()){
Toast.makeText(getApplicationContext(), "Searching for glasses...", Toast.LENGTH_LONG).show();
// EventBus.getDefault().post(new PostGenericGlobalMessageEvent("Searching for glasses..."));
Log.d(TAG, "TRYING TO CONNECT TO: " + smartGlassesDevices.get(0).deviceModelName);
if (smartGlassesRepresentative != null) {
smartGlassesRepresentative.destroy();
smartGlassesRepresentative = null;
}
connectToSmartGlasses(smartGlassesDevices.get(0));
smartGlassesDevices.add(smartGlassesDevices.remove(0));
aioRetryHandler.postDelayed(this, 5000); // Schedule another retry if needed
}
else
{
aioRetryHandler.removeCallbacks(this);
Toast.makeText(getApplicationContext(), "No glasses found", Toast.LENGTH_LONG).show();
// EventBus.getDefault().post(new PostGenericGlobalMessageEvent("No glasses found"));
}
}
else {
Toast.makeText(getApplicationContext(), "Connected to " + smartGlassesRepresentative.smartGlassesDevice.deviceModelName, Toast.LENGTH_LONG).show();
// EventBus.getDefault().post(new PostGenericGlobalMessageEvent("Connected to " + smartGlassesRepresentative.smartGlassesDevice.deviceModelName));
}
}
};
public void aioConnectSmartGlasses(){
if (getChosenAsrFramework(this) == ASR_FRAMEWORKS.GOOGLE_ASR_FRAMEWORK) {
String apiKey = getApiKey(getApplicationContext());
if (apiKey == null || apiKey.equals("")) {
showNoGoogleAsrDialog();
return;
}
}
String preferred = getPreferredWearable(this.getApplicationContext());
smartGlassesDevices = new ArrayList(Arrays.asList(new VuzixUltralite(), new VuzixShield(), new InmoAirOne(), new TCLRayNeoXTwo()));
for (int i = 0; i < smartGlassesDevices.size(); i++){
if (smartGlassesDevices.get(i).deviceModelName.equals(preferred)){
// Move to start for earliest search priority
smartGlassesDevices.add(0, smartGlassesDevices.remove(i));
break;
}
}
// Check for Audio Wearable
if (preferred.equals(new AudioWearable().deviceModelName))
smartGlassesDevices.add(0, new AudioWearable());
//start loop
aioRetryConnectionTask.run();
}
public void showNoGoogleAsrDialog(){
new android.app.AlertDialog.Builder(getApplicationContext()).setIcon(android.R.drawable.ic_dialog_alert)
.setTitle("No Google API Key Provided")
.setMessage("You have Google ASR enabled without an API key. Please turn off Google ASR or enter a valid API key.")
.setPositiveButton("Ok", new DialogInterface.OnClickListener() {
@Override
public void onClick(DialogInterface dialog, int which) {
}
}).show();
}
//show a reference card on the smart glasses with title and body text
public static void sendReferenceCard(String title, String body) {
EventBus.getDefault().post(new ReferenceCardSimpleViewRequestEvent(title, body));
}
//show a text wall card on the smart glasses
public static void sendTextWall(String text) {
EventBus.getDefault().post(new TextWallViewRequestEvent(text));
}
//show a double text wall card on the smart glasses
public static void sendDoubleTextWall(String textTop, String textBottom) {
EventBus.getDefault().post(new DoubleTextWallViewRequestEvent(textTop, textBottom));
}
//show a reference card on the smart glasses with title and body text
public static void sendRowsCard(String[] rowStrings) {
EventBus.getDefault().post(new RowsCardViewRequestEvent(rowStrings));
}
//show a bullet point list card on the smart glasses with title and bullet points
public void sendBulletPointList(String title, String [] bullets) {
EventBus.getDefault().post(new BulletPointListViewRequestEvent(title, bullets));
}
//show a list of up to 4 rows of text. Only put a few characters per line!
public void sendBulletPointList(String[] rowStrings) {
EventBus.getDefault().post(new RowsCardViewRequestEvent(rowStrings));
}
public void sendReferenceCard(String title, String body, String imgUrl) {
EventBus.getDefault().post(new ReferenceCardImageViewRequestEvent(title, body, imgUrl));
}
public void sendBitmap(Bitmap bitmap) {
EventBus.getDefault().post(new SendBitmapViewRequestEvent(bitmap));
}
public void startScrollingText(String title){
EventBus.getDefault().post(new ScrollingTextViewStartRequestEvent(title));
}
public void pushScrollingText(String text){
EventBus.getDefault().post(new FinalScrollingTextRequestEvent(text));
}
public void stopScrollingText(){
EventBus.getDefault().post(new ScrollingTextViewStopRequestEvent());
}
public void sendTextLine(String text) {
EventBus.getDefault().post(new TextLineViewRequestEvent(text));
}
public void sendTextToSpeech(String text, String languageString) {
EventBus.getDefault().post(new TextToSpeechEvent(text, languageString));
}
public void sendCenteredText(String text){
EventBus.getDefault().post(new CenteredTextViewRequestEvent(text));
}
public void sendCustomContent(String json){
EventBus.getDefault().post(new CenteredTextViewRequestEvent(json));
}
public void sendHomeScreen(){
EventBus.getDefault().post(new HomeScreenEvent());
}
public void setFontSize(SmartGlassesFontSize fontSize) { EventBus.getDefault().post(new SetFontSizeEvent(fontSize)); }
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/SmartGlassesRepresentative.java
================================================
package com.teamopensmartglasses.smartglassesmanager;
import android.content.Context;
import org.greenrobot.eventbus.EventBus;
import org.greenrobot.eventbus.Subscribe;
import org.json.JSONObject;
import android.os.Handler;
import android.util.Log;
//custom, our code
import androidx.lifecycle.LifecycleOwner;
import com.teamopensmartglasses.augmentoslib.events.AudioChunkNewEvent;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.DisableBleScoAudioEvent;
import com.teamopensmartglasses.augmentoslib.events.DisplayCustomContentRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.DoubleTextWallViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.HomeScreenEvent;
import com.teamopensmartglasses.augmentoslib.events.SendBitmapViewRequestEvent;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.SetFontSizeEvent;
import com.teamopensmartglasses.augmentoslib.events.TextWallViewRequestEvent;
import com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators.AudioWearableSGC;
import com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators.UltraliteSGC;
import com.teamopensmartglasses.augmentoslib.events.BulletPointListViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.CenteredTextViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.FinalScrollingTextRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.IntermediateScrollingTextRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.ReferenceCardImageViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.ReferenceCardSimpleViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.RowsCardViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.PromptViewRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.ScrollingTextViewStartRequestEvent;
import com.teamopensmartglasses.augmentoslib.events.ScrollingTextViewStopRequestEvent;
import com.teamopensmartglasses.smartglassesmanager.hci.AudioChunkCallback;
import com.teamopensmartglasses.smartglassesmanager.hci.MicrophoneLocalAndBluetooth;
//import com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators.ActiveLookSGC;
import com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators.AndroidSGC;
import com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators.SmartGlassesCommunicator;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.SmartGlassesDevice;
import com.teamopensmartglasses.augmentoslib.events.TextLineViewRequestEvent;
//rxjava
import java.nio.ByteBuffer;
import io.reactivex.rxjava3.subjects.PublishSubject;
class SmartGlassesRepresentative {
private static final String TAG = "WearableAi_ASGRepresentative";
//receive/send data stream
PublishSubject dataObservable;
Context context;
public SmartGlassesDevice smartGlassesDevice;
SmartGlassesCommunicator smartGlassesCommunicator;
MicrophoneLocalAndBluetooth bluetoothAudio;
//timing settings
long referenceCardDelayTime = 10000;
LifecycleOwner lifecycleOwner;
//handler to handle delayed UI events
Handler uiHandler;
Handler micHandler;
SmartGlassesRepresentative(Context context, SmartGlassesDevice smartGlassesDevice, LifecycleOwner lifecycleOwner, PublishSubject dataObservable){
this.context = context;
this.smartGlassesDevice = smartGlassesDevice;
this.lifecycleOwner = lifecycleOwner;
//receive/send data
this.dataObservable = dataObservable;
uiHandler = new Handler();
micHandler = new Handler();
//register event bus subscribers
EventBus.getDefault().register(this);
}
public void connectToSmartGlasses(){
switch (smartGlassesDevice.getGlassesOs()){
case ANDROID_OS_GLASSES:
smartGlassesCommunicator = new AndroidSGC(context, dataObservable);
break;
// case ACTIVELOOK_OS_GLASSES:
// smartGlassesCommunicator = new ActiveLookSGC(context);
// break;
case AUDIO_WEARABLE_GLASSES:
smartGlassesCommunicator = new AudioWearableSGC(context);
break;
case ULTRALITE_MCU_OS_GLASSES:
smartGlassesCommunicator = new UltraliteSGC(context, lifecycleOwner);
break;
case EVENREALITIES_G1_MCU_OS_GLASSES:
// smartGlassesCommunicator = new EvenRealitiesG1SGC(context, lifecycleOwner);
break;
}
smartGlassesCommunicator.connectToSmartGlasses();
//if the glasses don't support a microphone, this Representative handles local microphone
if (smartGlassesDevice.useScoMic) {
connectAndStreamLocalMicrophone(true);
} else if (!smartGlassesDevice.getHasInMic() && !smartGlassesDevice.getHasOutMic()) {
connectAndStreamLocalMicrophone(false);
}
}
@Subscribe
public void onDisableBleScoEvent(DisableBleScoAudioEvent receivedEvent) {
Log.d(TAG, "onDisableBleScoEvent called");
restartAudioWithNoBleSco();
}
public void restartAudioWithNoBleSco(){
//kill current audio
if (bluetoothAudio != null) {
bluetoothAudio.destroy();
}
//start new audio, with no bluetooth
connectAndStreamLocalMicrophone(false);
}
private void connectAndStreamLocalMicrophone(boolean useBluetoothSco){
//follow this order for speed
//start audio from bluetooth headset
uiHandler.post(new Runnable() {
@Override
public void run() {
bluetoothAudio = new MicrophoneLocalAndBluetooth(context, useBluetoothSco, new AudioChunkCallback(){
@Override
public void onSuccess(ByteBuffer chunk){
receiveChunk(chunk);
}
});
}
});
}
private void receiveChunk(ByteBuffer chunk){
byte[] audio_bytes = chunk.array();
//throw off new audio chunk event
EventBus.getDefault().post(new AudioChunkNewEvent(audio_bytes));
}
public void destroy(){
Log.d(TAG, "SG rep destroying");
EventBus.getDefault().unregister(this);
if (bluetoothAudio != null) {
bluetoothAudio.destroy();
}
if (smartGlassesCommunicator != null){
smartGlassesCommunicator.destroy();
smartGlassesCommunicator = null;
}
Log.d(TAG, "SG rep destroy complete");
}
//are our smart glasses currently connected?
public int getConnectionState(){
if (smartGlassesCommunicator == null){
return 0;
} else {
return smartGlassesCommunicator.getConnectionState();
}
}
public void showReferenceCard(String title, String body){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayReferenceCardSimple(title, body);
}
}
public void showRowsCard(String[] rowStrings){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayRowsCard(rowStrings);
}
}
public void startScrollingTextViewModeTest(){
//pass for now
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.startScrollingTextViewMode("ScrollingTextView");
smartGlassesCommunicator.scrollingTextViewFinalText("test line 1");
smartGlassesCommunicator.scrollingTextViewFinalText("line 2 testy boi");
smartGlassesCommunicator.scrollingTextViewFinalText("how's this?");
smartGlassesCommunicator.scrollingTextViewFinalText("this is a line of text that is going to be long enough to wrap around, it would be good to see if it doesn so, that would be super cool");
smartGlassesCommunicator.scrollingTextViewFinalText("test line n");
smartGlassesCommunicator.scrollingTextViewFinalText("line n + 1 testy boi");
smartGlassesCommunicator.scrollingTextViewFinalText("seconnndd how's this?");
}
}
private void homeUiAfterDelay(long delayTime){
uiHandler.postDelayed(new Runnable() {
@Override
public void run() {
homeScreen();
}
}, delayTime);
}
public void homeScreen(){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.showHomeScreen();
}
}
@Subscribe
public void onHomeScreenEvent(HomeScreenEvent receivedEvent){
homeScreen();
}
@Subscribe
public void onTextWallViewEvent(TextWallViewRequestEvent receivedEvent){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayTextWall(receivedEvent.text);
}
}
@Subscribe
public void onDoubleTextWallViewEvent(DoubleTextWallViewRequestEvent receivedEvent){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayDoubleTextWall(receivedEvent.textTop, receivedEvent.textBottom);
}
}
@Subscribe
public void onReferenceCardSimpleViewEvent(ReferenceCardSimpleViewRequestEvent receivedEvent){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayReferenceCardSimple(receivedEvent.title, receivedEvent.body);
// homeUiAfterDelay(referenceCardDelayTime);
}
}
@Subscribe
public void onRowsCardViewEvent(RowsCardViewRequestEvent receivedEvent){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayRowsCard(receivedEvent.rowStrings);
// homeUiAfterDelay(referenceCardDelayTime);
}
}
@Subscribe
public void onBulletPointListViewEvent(BulletPointListViewRequestEvent receivedEvent){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayBulletList(receivedEvent.title, receivedEvent.bullets);
// homeUiAfterDelay(referenceCardDelayTime);
}
}
@Subscribe
public void onReferenceCardImageViewEvent(ReferenceCardImageViewRequestEvent receivedEvent){
Log.d(TAG, "sending reference card image view event");
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayReferenceCardImage(receivedEvent.title, receivedEvent.body, receivedEvent.imgUrl);
// homeUiAfterDelay(referenceCardDelayTime);
}
}
@Subscribe
public void onSendBitmapViewRequestEvent(SendBitmapViewRequestEvent receievedEvent){
Log.d(TAG, "Sending a bitmap event");
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayBitmap(receievedEvent.bmp);
}
}
@Subscribe
public void onDisplayCustomContentRequestEvent(DisplayCustomContentRequestEvent receivedEvent){
Log.d(TAG, "Got display custom content event: " + receivedEvent.json);
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayCustomContent(receivedEvent.json);
}
}
@Subscribe
public void onTextLineViewRequestEvent(TextLineViewRequestEvent receivedEvent){
Log.d(TAG, "Got text line event: " + receivedEvent.text);
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayTextLine(receivedEvent.text);
}
}
@Subscribe
public void onDisplayCenteredTextRequestEvent(CenteredTextViewRequestEvent receivedEvent){
if(smartGlassesCommunicator != null){
smartGlassesCommunicator.displayCenteredText(receivedEvent.text);
}
}
@Subscribe
public void onStartScrollingTextViewEvent(ScrollingTextViewStartRequestEvent receivedEvent){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.startScrollingTextViewMode(receivedEvent.title);
}
}
@Subscribe
public void onStopScrollingTextViewEvent(ScrollingTextViewStopRequestEvent receivedEvent){
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.stopScrollingTextViewMode();
}
}
@Subscribe
public void onFinalScrollingTextEvent(FinalScrollingTextRequestEvent receivedEvent) {
Log.d(TAG, "onFinalScrollingTextEvent");
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.scrollingTextViewFinalText(receivedEvent.text);
}
}
@Subscribe
public void onIntermediateScrollingTextEvent(IntermediateScrollingTextRequestEvent receivedEvent) {
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.scrollingTextViewIntermediateText(receivedEvent.text);
}
}
@Subscribe
public void onPromptViewRequestEvent(PromptViewRequestEvent receivedEvent) {
Log.d(TAG, "onPromptViewRequestEvent called");
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.displayPromptView(receivedEvent.prompt, receivedEvent.options);
}
}
@Subscribe
public void onSetFontSizeEvent(SetFontSizeEvent receivedEvent) {
if (smartGlassesCommunicator != null) {
smartGlassesCommunicator.setFontSize(receivedEvent.fontSize);
}
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/comms/AspWebsocketServer.java
================================================
package com.teamopensmartglasses.smartglassesmanager.comms;
import java.net.InetSocketAddress;
import org.java_websocket.WebSocket;
import org.java_websocket.exceptions.WebsocketNotConnectedException;
import org.java_websocket.handshake.ClientHandshake;
import org.java_websocket.server.WebSocketServer;
import java.util.Map;
import java.util.UUID;
import java.nio.ByteBuffer;
import java.util.concurrent.ConcurrentHashMap;
import android.util.Log;
import org.json.JSONObject;
import org.json.JSONException;
import io.reactivex.rxjava3.disposables.Disposable;
import io.reactivex.rxjava3.subjects.PublishSubject;
public class AspWebsocketServer extends WebSocketServer {
//data observable we can send data through
private static PublishSubject dataObservable;
private static Disposable dataSub;
private static int connected = 0;
private final String TAG = "WearableAI_AspWebsocketServer";
private Map clients = new ConcurrentHashMap<>();
private WebSocket asgConn;
public AspWebsocketServer(int port)
{
super(new InetSocketAddress(port));
setReuseAddr(true);
}
public AspWebsocketServer(InetSocketAddress address)
{
super(address);
}
@Override
public void onOpen(WebSocket conn, ClientHandshake handshake) {
connected = 2;
String uniqueID = UUID.randomUUID().toString();
Log.d(TAG, "Got new connection at address: " + conn.getRemoteSocketAddress());
clients.put(uniqueID, conn);
asgConn = conn;
}
@Override
public void onClose(WebSocket conn, int code, String reason, boolean remote) {
Log.d(TAG, "onClose called");
Log.d(TAG, "WebSocketServer currently has this many conns: " + clients.size());
for (WebSocket connToCheck : clients.values()){ // there was a race condition where the ASG would recconnect before the socket was marked as closed, so the ASP thought their was no connection (connected = 1), but there actually was a connection, so now we check to see if any of the sockets in the clients hashmap is still open
Log.d(TAG, "Checking conn with remoteSocketAddress: " + connToCheck.getRemoteSocketAddress());
if (connToCheck.isOpen()){ //exit, because we have a live connection to the ASG
Log.d(TAG, "onClose found open connection, so not setting connected=1");
return;
}
}
Log.d(TAG, "onClose set connected=1");
clients.clear();
connected = 1;
}
@Override
public void onMessage(WebSocket conn, String message) {
try {
// Log.d(TAG, message);
JSONObject json_obj = new JSONObject(message);
dataObservable.onNext(json_obj);
} catch (JSONException e){
//if we send a string, this will get thrown, all messages should be JSON or byte []
// e.printStackTrace();
}
}
@Override
public void onMessage(WebSocket conn, ByteBuffer message) {
Log.d(TAG, "GOT MESSAGE BYTES");
}
@Override
public void onError(WebSocket conn, Exception ex)
{
ex.printStackTrace();
if (conn != null) {
// some errors like port binding failed may not be assignable to a specific websocket
}
}
@Override
public void onStart()
{
//LogHelper.e(TAG, "Server started!");
connected = 1;
setConnectionLostTimeout(2);
startConnectionLostTimer();
}
public void sendJson(JSONObject data){
if (connected == 2){
try {
asgConn.send(data.toString());
} catch (WebsocketNotConnectedException e){
e.printStackTrace();
Log.d(TAG, "Not connected, close should be running now.");
}
} else {
Log.d(TAG, "CANNOT SEND JSON, NOT CONNECTED");
}
}
//receive observable to send and receive data
public void setObservable(PublishSubject observable){
Log.d(TAG, "called setObservable");
dataObservable = observable;
dataSub = dataObservable.subscribe(i -> handleDataStream(i));
}
//this receives data from the data observable. For now, this class decides what to send and what not to send to the ASG
private void handleDataStream(JSONObject data){
//first check if it's a type we should handle
try{
String type = data.getString(MessageTypes.MESSAGE_TYPE_LOCAL);
// if (type.equals(MessageTypes.INTERMEDIATE_TRANSCRIPT)){
//// Log.d(TAG, "AspWebsocketServer got INTERMEDIATE_TRANSCRIPT, sending to ASG");
// //data.put(MessageTypes.MESSAGE_TYPE_LOCAL, data.getString(MessageTypes.MESSAGE_TYPE_ASG)); //change the type to the type for ASG
// //data.remove(MessageTypes.MESSAGE_TYPE_ASG);
// sendJson(data);
// } else if (type.equals(MessageTypes.FINAL_TRANSCRIPT)){
//// Log.d(TAG, "AspWebsocketServer got FINAL_TRANSCRIPT, sending to ASG");
// sendJson(data);
if (type.equals(MessageTypes.VOICE_COMMAND_RESPONSE)){
// Log.d(TAG, "AspWebsocketServer got VOICE_COMMAND_RESPONSE, sending to ASG");
sendJson(data);
} else if (type.equals(MessageTypes.REFERENCE_CARD_SIMPLE_VIEW)){
// Log.d(TAG, "AspWebsocketServer got REFERENCE_CARD_SIMPLE_VIEW, sending to ASG");
sendJson(data);
} else if (type.equals(MessageTypes.REFERENCE_CARD_TEXT_WALL_VIEW)){
// Log.d(TAG, "AspWebsocketServer got REFERENCE_CARD_SIMPLE_VIEW, sending to ASG");
sendJson(data);
} else if (type.equals(MessageTypes.REFERENCE_CARD_IMAGE_VIEW)){
sendJson(data);
} else if (type.equals(MessageTypes.ACTION_SWITCH_MODES)){
// Log.d(TAG, "AspWebsocketServer got ACTION_SWITCH_MODES, sending to ASG");
sendJson(data);
} else if (type.equals(MessageTypes.SCROLLING_TEXT_VIEW_START)){
// Log.d(TAG, "AspWebsocketServer sending STV start");
sendJson(data);
} else if (type.equals(MessageTypes.SCROLLING_TEXT_VIEW_STOP)){
// Log.d(TAG, "AspWebsocketServer sending STV start");
sendJson(data);
} else if (type.equals(MessageTypes.SCROLLING_TEXT_VIEW_FINAL)){
// Log.d(TAG, "AspWebsocketServer sending STV final");
sendJson(data);
} else if (type.equals(MessageTypes.SCROLLING_TEXT_VIEW_INTERMEDIATE)){
// Log.d(TAG, "AspWebsocketServer sending STV intermediate");
sendJson(data);
} else if (type.equals(MessageTypes.VOICE_COMMAND_STREAM_EVENT)) {
// Log.d(TAG, "AspWebsocketServer got VOICE_COMMAND_STREAM_EVENT, sending to ASG");
sendJson(data);
}
} catch (JSONException e){
e.printStackTrace();
}
}
//need to call this so if we get "Force Stop"ped, we will clean up sockets so we can connect on restart
public void destroy(){
Log.d(TAG, "destroying");
connected = 0;
dataSub.dispose();
dataSub = null;
try{
stop(400);
} catch (InterruptedException e){
e.printStackTrace();
}
Log.d(TAG, "destroy complete");
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/comms/AudioSystem.java
================================================
package com.teamopensmartglasses.smartglassesmanager.comms;
import android.media.AudioFormat;
import android.util.Base64;
import io.reactivex.rxjava3.disposables.Disposable;
import io.reactivex.rxjava3.subjects.PublishSubject;
import java.io.IOException;
import org.greenrobot.eventbus.EventBus;
import org.json.JSONObject;
import org.json.JSONException;
import android.content.Context;
import java.util.Random;
import java.nio.ByteOrder;
import android.os.Handler;
import android.os.HandlerThread;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.io.DataOutputStream;
import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.io.DataInputStream;
import java.net.ServerSocket;
import java.net.Socket;
import com.teamopensmartglasses.augmentoslib.events.AudioChunkNewEvent;
import com.teamopensmartglasses.smartglassesmanager.utils.AES;
import android.util.Log;
import com.teamopensmartglasses.smartglassesmanager.R;
public class AudioSystem {
private static String TAG = "WearableAi_AudioSystem";
private boolean shouldDie;
private String secretKey;
// the audio recording options - same on ASG
private static final int RECORDING_RATE = 16000;
private static final int CHANNEL = AudioFormat.CHANNEL_IN_MONO;
private static final int FORMAT = AudioFormat.ENCODING_PCM_16BIT;
//Thread to receive audio from ASG
//socket info
static int PORT = 4449;
private static int mConnectState = 0;
final byte [] ack_id = {0x13, 0x37};
final byte [] heart_beat_id = {0x19, 0x20};
final byte [] img_id = {0x01, 0x10}; //id for images
//handle heart beat stuff
private static long lastHeartbeatTime;
private static int heartbeatInterval = 3000; //milliseconds
private static int heartbeatPanicX = 3; // number of intervals before we reset connection
static Thread HeartbeatThread = null;
private int outbound_heart_beats = 0;
//socket data
static Thread SocketThread = null;
static Thread ReceiveThread = null;
static Thread SendThread = null;
//i/o
private DataOutputStream output;
private DataInputStream input;
//our actual socket connection object
ServerSocket serverSocket;
private static Socket socket;
//send audio to to other services in the app
PublishSubject dataObservable;
Disposable dataSubscriber;
Context context;
public AudioSystem(Context context, PublishSubject dataObservable){
this.context = context;
//set the key for encryption
secretKey = context.getResources().getString(R.string.key);
this.dataObservable = dataObservable;
dataSubscriber = dataObservable.subscribe(i -> handleDataStream(i));
}
//send_queue of data to send through the socket
private BlockingQueue send_queue;
public void startAudio(){
//make a new queue to hold data to send
send_queue = new ArrayBlockingQueue(50);
//start the socket thread which will send the raw audio data
startSocket();
}
public void startSocket(){
//start first socketThread
if (socket == null) {
mConnectState = 1;
SocketThread = new Thread(new SocketThread());
SocketThread.start();
//setup handler to handle keeping connection alive, all subsequent start of SocketThread
//start a new handler thread to send heartbeats
HandlerThread thread = new HandlerThread("HeartBeater");
thread.start();
Handler heart_beat_handler = new Handler(thread.getLooper());
final int hb_delay = 3000;
final int min_hb_delay = 1000;
final int max_hb_delay = 2000;
Random rand = new Random();
heart_beat_handler.postDelayed(new Runnable() {
public void run() {
heartBeat();
//random hb_delay for heart beat so as to disallow synchronized failure between client and server
int random_hb_delay = rand.nextInt((max_hb_delay - min_hb_delay) + 1) + min_hb_delay;
heart_beat_handler.postDelayed(this, random_hb_delay);
}
}, hb_delay);
}
}
//heart beat checker - check if we have received a heart rate
// private void heartBeat(){
// //check if we are still connected.
// //if not , reconnect,
// //we don't need to actively send heart beats from the client, as it's assumed that we are ALWAYS streaming data. Later, if we have periods of time where no data is sent, we will want to send a heart beat perhaps. but the client doesn't really need to, we just need to check if we are still connected
// if (mConnectState == 0) {
// restartSocket();
// }
//
// //or, if haven't been receiving heart beats, restart socket
// if (mConnectState == 2) {
// if ((System.currentTimeMillis() - lastHeartbeatTime) > (heartbeatInterval * heartbeatPanicX)) {
// Log.d(TAG, "DIDN'T RECEIVE HEART BEATS, RESTARTING SOCKET");
// mConnectState = 0;
// restartSocket();
// }
// }
// }
//
private void heartBeat(){
//check if we are still connected.
//if not , reconnect,
//if we are connected, send a heart beat to make sure we are still connected
if ((mConnectState == 0) && (shouldDie == false)) {
restartSocket();
} else if (mConnectState == 2){
//make sure we don't have a ton of outbound heart beats unresponded to
//reimplement this later -- ASG needs to receive heart beats
// if (outbound_heart_beats > 5) {
// restartSocket();
// return;
// }
//
// //increment counter
// outbound_heart_beats++;
//
// //send heart beat
// sendBytes(heart_beat_id, null);
}
}
// public static void restartSocket() {
// Log.d(TAG, "Restarting socket");
// mConnectState = 1;
// if (socket != null && (!socket.isClosed())){
// try {
// output.close();
// input.close();
// socket.close();
// } catch (IOException e) {
// System.out.println("FAILED TO CLOSE SOCKET, SOMETHING IS WRONG");
// }
// }
//
//
// //restart socket thread
// Log.d(TAG, "starting socket");
// SocketThread = new Thread(new SocketThread());
// SocketThread.start();
// }
//
private void restartSocket(){
Log.d(TAG, "Running restart socket");
mConnectState = 1;
outbound_heart_beats = 0;
//close the previous socket now that it's broken/being restarted
killSocket();
//make sure socket thread has joined before throwing off a new one
try {
Log.d(TAG, "Waiting socket thread join");
SocketThread.join();
Log.d(TAG, "Socket thread joined");
} catch (InterruptedException e){
e.printStackTrace();
}
//start a new socket thread
SocketThread = new Thread(new SocketThread());
SocketThread.start();
}
private void killSocket(){
try {
if (serverSocket != null && (!serverSocket.isClosed())) {
Log.d(TAG, "Closing socket, input, serverSocket, etc.");
serverSocket.close();
}
if (socket != null){
socket.close();
}
if (output != null){
output.close();
}
if (input != null){
input.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
// static class SocketThread implements Runnable {
// public void run() {
// try {
// serverSocket = new ServerSocket(PORT);
// try {
// socket = serverSocket.accept();
// // System.out.println("TRYING TO CONNECT AUDIO STREAM ASG");
// // socket = new Socket(SERVER_IP, SERVER_PORT);
// lastHeartbeatTime = System.currentTimeMillis();
// System.out.println("GLBOX CONNECTED!");
// //output = new DataOutputStream(socket.getOutputStream());
// //input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
// input = new DataInputStream(new DataInputStream(socket.getInputStream()));
// mConnectState = 2;
// //make the threads that will send and receive
// if (ReceiveThread == null) { //if the thread is null, make a new one (the first one)
// ReceiveThread = new Thread(new ReceiveThread());
// ReceiveThread.start();
// } else if (!ReceiveThread.isAlive()) { //if the thread is not null but it's dead, let it join then start a new one
// Log.d(TAG, "IN SocketThread, WAITING FOR receive THREAD JOING");
// try {
// ReceiveThread.join(); //make sure socket thread has joined before throwing off a new one
// } catch (InterruptedException e) {
// e.printStackTrace();
// }
// Log.d(TAG, "receive JOINED");
// ReceiveThread = new Thread(new ReceiveThread());
// ReceiveThread.start();
// }
// } catch(IOException e){
// e.printStackTrace();
// mConnectState = 0;
// }
// } catch (IOException e) {
// Log.d(TAG, "Connection Refused on socket");
// e.printStackTrace();
// mConnectState = 0;
// }
// }
// }
//
class SocketThread implements Runnable {
@Override
public void run() {
try {
Log.d(TAG, "Starting new socket, waiting for connection...");
serverSocket = new ServerSocket(PORT);
try {
socket = serverSocket.accept();
socket.setSoTimeout(3000);
Log.d(TAG, "Got socket connection.");
//output = new PrintWriter(socket.getOutputStream(), true);
output = new DataOutputStream(socket.getOutputStream());
input = new DataInputStream(new DataInputStream(socket.getInputStream()));
mConnectState = 2;
if (ReceiveThread == null) { //if the thread is null, make a new one (the first one)
ReceiveThread = new Thread(new ReceiveThread());
ReceiveThread.start();
} else if (!ReceiveThread.isAlive()) { //if the thread is not null but it's dead, let it join then start a new one
try {
ReceiveThread.join(); //make sure socket thread has joined before throwing off a new one
} catch (InterruptedException e) {
e.printStackTrace();
}
ReceiveThread = new Thread(new ReceiveThread());
ReceiveThread.start();
}
if (SendThread == null) { //if the thread is null, make a new one (the first one)
SendThread = new Thread(new SendThread());
SendThread.start();
} else if (!SendThread.isAlive()) { //if the thread is not null but it's dead, let it join then start a new one
try {
SendThread.join(); //make sure socket thread has joined before throwing off a new one
} catch (InterruptedException e) {
e.printStackTrace();
}
SendThread = new Thread(new SendThread());
SendThread.start();
}
} catch (IOException e) {
e.printStackTrace();
mConnectState = 0;
}
} catch (IOException e) {
e.printStackTrace();
mConnectState = 0;
}
}
}
public void sendBytes(byte[] id, byte [] data){
//first, send hello
byte [] hello = {0x01, 0x02, 0x03};
//then send length of body
byte[] len;
if (data != null) {
len = my_int_to_bb_be(data.length);
} else {
len = my_int_to_bb_be(0);
}
//then send id of message type
byte [] msg_id = id;
//then send data
byte [] body = data;
//then send end tag - eventually make this unique to the image
byte [] goodbye = {0x3, 0x2, 0x1};
//combine those into a payload
ByteArrayOutputStream outputStream;
try {
outputStream = new ByteArrayOutputStream();
outputStream.write(hello);
outputStream.write(len);
outputStream.write(msg_id);
if (body != null) {
outputStream.write(body);
}
outputStream.write(goodbye);
} catch (IOException e){
mConnectState = 0;
return;
}
byte [] payload = outputStream.toByteArray();
//send it in a background thread
//new Thread(new SendThread(payload)).start();
send_queue.add(payload);
}
//this sends messages
class SendThread implements Runnable {
SendThread() {
}
@Override
public void run() {
send_queue.clear();
while (true){
if (mConnectState != 2){
break;
}
if (send_queue.size() > 10){
break;
}
byte [] data;
try {
data = send_queue.take(); //block until there is something we can pull out to send
} catch (InterruptedException e){
e.printStackTrace();
break;
}
try {
output.write(data); // write the message
} catch (java.io.IOException e) {
e.printStackTrace();
break;
}
}
throwBrokenSocket();
}
}
//receives messages
private class ReceiveThread implements Runnable {
@Override
public void run() {
//System.out.println("Receive Started, mconnect: " + mConnectState);
while (true) {
if (mConnectState != 2){
break;
}
try {
int chunk_len = 6416; //until we use a better protocol to specify start and end of packet, we need to to match the number in asg
byte [] raw_data = new byte[chunk_len];
input.readFully(raw_data, 0, chunk_len); // read the body
EventBus.getDefault().post(new AudioChunkNewEvent(raw_data));
//byte [] plain_audio_bytes = decryptBytes(raw_data);
//dataObservable.onNext(plain_audio_bytes);
} catch (IOException e) {
Log.d(TAG, "Audio service receive thread broken.");
e.printStackTrace();
break;
}
}
throwBrokenSocket();
}
}
public byte[] my_int_to_bb_be(int myInteger){
return ByteBuffer.allocate(4).order(ByteOrder.BIG_ENDIAN).putInt(myInteger).array();
}
private void throwBrokenSocket(){
if (mConnectState == 2){
mConnectState = 0;
}
}
public byte [] decryptBytes(byte [] input) {
byte [] decryptedBytes = AES.decrypt(input, secretKey);
return decryptedBytes;
}
public void destroy(){
shouldDie = true;
dataSubscriber.dispose();
killSocket();
}
private void handleDataStream(JSONObject data){
try {
String dataType = data.getString(MessageTypes.MESSAGE_TYPE_LOCAL);
if (dataType.equals(MessageTypes.AUDIO_CHUNK_ENCRYPTED)) {
handleEncryptedData(data);
} else if (dataType.equals(MessageTypes.AUDIO_CHUNK_DECRYPTED)){
String encodedPlainData = data.getString(MessageTypes.AUDIO_DATA);
byte [] decodedPlainData = Base64.decode(encodedPlainData, Base64.DEFAULT);
EventBus.getDefault().post(new AudioChunkNewEvent(decodedPlainData));
}
} catch (JSONException e){
e.printStackTrace();
}
}
//here we decode, decrypt, the encode again. It's not pretty, but it allows us to use a JSON event bus, which makes things way more manageable and modular
private void handleEncryptedData(JSONObject data){
try{
String encodedData = data.getString(MessageTypes.AUDIO_DATA);
byte [] decodedData = Base64.decode(encodedData, Base64.DEFAULT);
byte [] plainData = decryptBytes(decodedData);
String encodedPlainData = Base64.encodeToString(plainData, Base64.DEFAULT);
//make new object and send as decrypted data
// JSONObject decryptedData = new JSONObject();
// decryptedData.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.AUDIO_CHUNK_DECRYPTED);
// decryptedData.put(MessageTypes.AUDIO_DATA, encodedPlainData);
// dataObservable.onNext(decryptedData);
//throw new audio event
EventBus.getDefault().post(new AudioChunkNewEvent(plainData));
} catch (JSONException e){
e.printStackTrace();
}
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/comms/MessageTypes.java
================================================
package com.teamopensmartglasses.smartglassesmanager.comms;
public class MessageTypes {
//top level
public static final String MESSAGE_TYPE_LOCAL = "MESSAGE_TYPE_LOCAL";
//SERVICE TO ACTIVITY
public static final String GLASSES_STATUS_UPDATE = "GLASSES_STATUS_UPDATE";
public static final String CONNECTION_GLASSES_STATUS_UPDATE = "CONNECTION_GLASSES_STATUS_UPDATE";
public static final String CONNECTION_GLASSES_GLASSES_OBJECT = "CONNECTION_GLASSES_GLASSES_OBJECT";
//REFERENCE CARD VIEW
public static final String REFERENCE_CARD_SIMPLE_VIEW = "REFERENCE_CARD_SIMPLE_VIEW";
public static final String REFERENCE_CARD_TEXT_WALL_VIEW = "REFERENCE_CARD_TEXT_WALL_VIEW";
public static final String REFERENCE_CARD_SIMPLE_VIEW_TITLE = "REFERENCE_CARD_SIMPLE_VIEW_TITLE";
public static final String REFERENCE_CARD_SIMPLE_VIEW_BODY = "REFERENCE_CARD_SIMPLE_VIEW_BODY";
public static final String REFERENCE_CARD_IMAGE_VIEW = "REFERENCE_CARD_IMAGE_VIEW";
public static final String REFERENCE_CARD_IMAGE_VIEW_TITLE = "REFERENCE_CARD_IMAGE_VIEW_TITLE";
public static final String REFERENCE_CARD_IMAGE_VIEW_BODY = "REFERENCE_CARD_IMAGE_VIEW_BODY";
public static final String REFERENCE_CARD_TEXT_WALL_TEXT = "REFERENCE_CARD_TEXT_WALL_TEXT";
public static final String REFERENCE_CARD_IMAGE_VIEW_IMG_URL = "REFERENCE_CARD_IMAGE_VIEW_IMG_URL";
//SCROLLING TEXT VIEW
public static final String SCROLLING_TEXT_VIEW_START = "SCROLLING_TEXT_VIEW_START";
public static final String SCROLLING_TEXT_VIEW_FINAL = "SCROLLING_TEXT_VIEW_FINAL";
public static final String SCROLLING_TEXT_VIEW_INTERMEDIATE = "SCROLLING_TEXT_VIEW_INTERMEDIATE";
public static final String SCROLLING_TEXT_VIEW_TEXT = "SCROLLING_TEXT_VIEW_TEXT";
public static final String SCROLLING_TEXT_VIEW_TITLE = "SCROLLING_TEXT_VIEW_TITLE";
public static final String SCROLLING_TEXT_VIEW_STOP = "SCROLLING_TEXT_VIEW_STOP";
//GLASSES CONNECTION EVENT/STATE
public static final int CONNECTED_GLASSES = 2;
public static final int DISCONNECTED_GLASSES = 0;
public static final int TRYNA_CONNECT_GLASSES = 1;
public static final int FAILED_CONNECTING_GLASSES = 3;
//DATA TYPES
public static final String POV_IMAGE = "POV_IMAGE";
public static final String JPG_BYTES_BASE64 = "JPG_BYTES_BASE64";
public static final String IMAGE_ID = "IMAGE_ID";
//specific message types (after MESSAGE_TYPE_*)
//TRANSCRIPTS
public static final String FINAL_TRANSCRIPT = "FINAL_TRANSCRIPT";
public static final String INTERMEDIATE_TRANSCRIPT = "INTERMEDIATE_TRANSCRIPT";
public static final String FINAL_TRANSCRIPT_FOREIGN = "FINAL_TRANSCRIPT_FOREIGN";
public static final String INTERMEDIATE_TRANSCRIPT_FOREIGN = "INTERMEDIATE_TRANSCRIPT_FOREIGN";
public static final String TRANSCRIPT_TEXT = "TRANSCRIPT_TEXT";
public static final String TRANSCRIPT_ID = "TRANSCRIPT_ID";
public static final String TIMESTAMP = "TIMESTAMP";
//VOICE COMMANDS
public static final String VOICE_COMMAND_RESPONSE = "VOICE_COMMAND_RESPONSE";
public static final String COMMAND_RESULT = "COMMAND_RESULT";
public static final String COMMAND_NAME = "COMMAND_NAME";
public static final String COMMAND_RESPONSE_DISPLAY_STRING = "COMMAND_RESPONSE_DISPLAY_STRING";
//voice command event
public static final String VOICE_COMMAND_STREAM_EVENT = "VOICE_COMMAND_STREAM_EVENT";
public static final String VOICE_COMMAND_STREAM_EVENT_TYPE = "VOICE_COMMAND_STREAM_EVENT_TYPE";
public static final String WAKE_WORD_EVENT_TYPE = "WAKE_WORD_EVENT_TYPE";
public static final String COMMAND_EVENT_TYPE = "COMMAND_EVENT_TYPE";
public static final String CANCEL_EVENT_TYPE = "CANCEL_EVENT_TYPE";
public static final String RESOLVE_EVENT_TYPE = "RESOLVE_EVENT_TYPE";
public static final String TEXT_RESPONSE_EVENT_TYPE = "TEXT_RESPONSE_EVENT_TYPE";
public static final String COMMAND_ARGS_EVENT_TYPE = "COMMAND_ARGS_EVENT_TYPE";
public static final String REQUIRED_ARG_EVENT_TYPE = "REQUIRED_ARG_EVENT_TYPE";
public static final String ARG_NAME = "ARG_NAME";
public static final String ARG_OPTIONS = "ARG_OPTIONS";
public static final String INPUT_VOICE_STRING = "INPUT_VOICE_STRING";
public static final String VOICE_ARG_EXPECT_TYPE = "VOICE_ARG_EXPECT_TYPE";
public static final String VOICE_ARG_EXPECT_NATURAL_LANGUAGE = "VOICE_ARG_EXPECT_NATURAL_LANGUAGE";
public static final String VOICE_COMMAND_LIST = "VOICE_COMMAND_LIST";
public static final String INPUT_WAKE_WORD = "INPUT_WAKE_WORD";
public static final String INPUT_VOICE_COMMAND_NAME = "INPUT_VOICE_COMMAND_NAME";
//FACE/PERSON SIGHTING
public static final String FACE_SIGHTING_EVENT = "FACE_SIGHTING_EVENT";
public static final String FACE_NAME = "FACE_NAME";
//SMS
public static final String SMS_REQUEST_SEND = "SMS_REQUEST_SEND";
public static final String SMS_MESSAGE_TEXT = "SMS_MESSAGE_TEXT";
public static final String SMS_PHONE_NUMBER = "SMS_PHONE_NUMBER";
//AUDIO
public static final String AUDIO_CHUNK_ENCRYPTED = "AUDIO_CHUNK_ENCRYPTED";
public static final String AUDIO_CHUNK_DECRYPTED = "AUDIO_CHUNK_DECRYPTED";
public static final String AUDIO_DATA = "AUDIO_DATA";
//AUTOCITER/WEARABLE-REFERENCER
public static final String AUTOCITER_START = "AUTOCITER_START";
public static final String AUTOCITER_STOP = "AUTOCITER_STOP";
public static final String AUTOCITER_PHONE_NUMBER = "AUTOCITER_PHONE_NUMBER";
public static final String AUTOCITER_POTENTIAL_REFERENCES = "AUTOCITER_POTENTIAL_REFERENCES";
public static final String AUTOCITER_REFERENCE_DATA = "AUTOCITER_REFERENCE_DATA";
//request user UI to display a list of possible choices to dipslay
public static final String REFERENCE_SELECT_REQUEST = "REFERENCE_SELECT_REQUEST";
public static final String REFERENCES = "REFERENCES";
//command responses to show
//Natural language
public final static String NATURAL_LANGUAGE_QUERY = "NATURAL_LANGUAGE_QUERY";
public final static String TEXT_QUERY = "TEXT_QUERY";
//visual search
public final static String VISUAL_SEARCH_RESULT = "VISUAL_SEARCH_RESULT"; //this is the ASG facing term
public final static String VISUAL_SEARCH_IMAGE= "VISUAL_SEARCH_IMAGE";
public final static String VISUAL_SEARCH_QUERY = "VISUAL_SEARCH_QUERY"; //this is the glbox facing term
public final static String VISUAL_SEARCH_DATA = "VISUAL_SEARCH_DATA"; //this is the payload
//search engine
public final static String SEARCH_ENGINE_QUERY = "SEARCH_ENGINE_QUERY";
public final static String SEARCH_ENGINE_RESULT = "SEARCH_ENGINE_RESULT";
public final static String SEARCH_ENGINE_RESULT_DATA = "SEARCH_ENGINE_RESULT_DATA";
public final static String SEARCH_ENGINE_RESULT_TITLE = "SEARCH_ENGINE_RESULT_TITLE";
public final static String SEARCH_ENGINE_RESULT_BODY = "SEARCH_ENGINE_RESULT_BODY";
public final static String SEARCH_ENGINE_RESULT_IMAGE = "SEARCH_ENGINE_RESULT_IMAGE";
//translation
public final static String TRANSLATE_TEXT_QUERY = "TRANSLATE_TEXT_QUERY";
public final static String TRANSLATE_TEXT_DATA = "TRANSLATE_TEXT_DATA";
public final static String TRANSLATE_TEXT_RESULT = "TRANSLATE_TEXT_RESULT";
public final static String TRANSLATE_TEXT_RESULT_DATA = "TRANSLATION_RESULT_DATA";
public final static String START_FOREIGN_LANGUAGE_ASR = "START_FOREIGN_LANGUAGE_ASR";
public final static String STOP_FOREIGN_LANGUAGE_ASR = "STOP_FOREIGN_LANGUAGE_ASR";
public final static String START_FOREIGN_LANGUAGE_SOURCE_LANGUAGE_NAME = "START_FOREIGN_LANGUAGE_SOURCE_LANGUAGE_NAME";
//contextual/semantic search
public final static String START_CONTEXTUAL_SEARCH = "START_CONTEXTUAL_SEARCH";
public final static String STOP_CONTEXTUAL_SEARCH = "STOP_CONTEXTUAL_SEARCH";
public final static String CONTEXTUAL_SEARCH_REQUEST = "CONTEXTUAL_SEARCH_REQUEST";
public final static String CONTEXTUAL_SEARCH_RESULT = "CONTEXTUAL_SEARCH_RESULT";
public final static String CONTEXTUAL_SEARCH_RESULT_DATA = "CONTEXTUAL_SEARCH_RESULT_DATA";
//object translation
public final static String START_OBJECT_DETECTION = "START_OBJECT_DETECTION";
public final static String STOP_OBJECT_DETECTION = "STOP_OBJECT_DETECTION";
public final static String OBJECT_TRANSLATION_REQUEST = "OBJECT_TRANSLATION_REQUEST";
public final static String OBJECT_TRANSLATION_RESULT = "OBJECT_TRANSLATION_RESULT";
public final static String OBJECT_TRANSLATION_RESULT_DATA = "OBJECT_TRANSLATION_RESULT_DATA";
public final static String AFFECTIVE_SUMMARY_RESULT = "AFFECTIVE_SUMMARY_RESULT";
public final static String COMMAND_SWITCH_MODE = "COMMAND_SWITCH_MODE";
//select command
public final static String ACTION_SELECT_COMMAND = "ACTION_SELECT_COMMAND";
public final static String SELECTION = "SELECTION";
//translate
public static final String REFERENCE_TRANSLATE_SEARCH_QUERY = "REFERENCE_TRANSLATE_SEARCH_QUERY";
public static final String REFERENCE_TRANSLATE_DATA = "REFERENCE_TRANSLATE_DATA";
public static final String REFERENCE_TRANSLATE_TARGET_LANGUAGE_CODE = "REFERENCE_TRANSLATE_TARGET_LANGUAGE_CODE";
public static final String REFERENCE_TRANSLATE_RESULT = "REFERENCE_TRANSLATE_RESULT";
public static final String REFERENCE_TRANSLATE_RESULT_DATA = "REFERENCE_TRANSLATE_RESULT_DATA";
//text to speech
public static final String TEXT_TO_SPEECH_SPEAK = "TEXT_TO_SPEECH_SPEAK";
public static final String TEXT_TO_SPEECH_SPEAK_DATA = "TEXT_TO_SPEECH_SPEAK_DATA";
public static final String TEXT_TO_SPEECH_TARGET_LANGUAGE_CODE = "TEXT_TO_SPEECH_TARGET_LANGUAGE_CODE";
//control the current mode of the ASG
public final static String ACTION_SWITCH_MODES = "ACTION_SWITCH_MODES";
public final static String NEW_MODE = "NEW_MODE";
public final static String MODE_HOME = "MODE_HOME";
public final static String MODE_VISUAL_SEARCH = "MODE_VISUAL_SEARCH";
public final static String MODE_LIVE_LIFE_CAPTIONS = "MODE_LIVE_LIFE_CAPTIONS";
public final static String MODE_SOCIAL_MODE = "MODE_SOCIAL_MODE";
public final static String MODE_CONVERSATION_MODE = "MODE_CONVERSATION_MODE";
public final static String MODE_CONTEXTUAL_SEARCH = "MODE_CONTEXTUAL_SEARCH";
public final static String MODE_REFERENCE_GRID = "MODE_REFERENCE_GRID";
public final static String MODE_WEARABLE_FACE_RECOGNIZER = "MODE_WEARABLE_FACE_RECOGNIZER";
public final static String MODE_LANGUAGE_TRANSLATE = "MODE_LANGUAGE_TRANSLATE";
public final static String MODE_OBJECT_TRANSLATE = "MODE_OBJECT_TRANSLATE";
public final static String MODE_TEXT_LIST = "MODE_TEXT_LIST";
public final static String MODE_TEXT_BLOCK = "MODE_TEXT_BLOCK";
public final static String MODE_BLANK = "MODE_BLANK";
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/comms/VolleyCallback.java
================================================
package com.teamopensmartglasses.smartglassesmanager.comms;
import org.json.JSONObject;
public interface VolleyCallback {
void onSuccess(JSONObject result);
void onFailure();
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/AudioChunkNewEvent.java
================================================
package com.teamopensmartglasses.augmentoslib.events;
public class AudioChunkNewEvent {
public byte [] thisChunk;
public AudioChunkNewEvent(byte [] thisChunk){
this.thisChunk = thisChunk;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/DisableBleScoAudioEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
public class DisableBleScoAudioEvent {
public boolean disableBleScoAudio;
public DisableBleScoAudioEvent(boolean disableBleScoAudio){
this.disableBleScoAudio = disableBleScoAudio;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/GoogleAudioChunkNewEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
public class GoogleAudioChunkNewEvent {
public byte [] thisChunk;
public GoogleAudioChunkNewEvent(byte [] thisChunk){
this.thisChunk = thisChunk;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/PauseAsrEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
public class PauseAsrEvent {
public boolean pauseAsr;
public PauseAsrEvent(boolean pauseAsr){
this.pauseAsr = pauseAsr;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/PostGenericGlobalMessageEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
public class PostGenericGlobalMessageEvent {
public String message;
public PostGenericGlobalMessageEvent(String newMessage){
this.message = newMessage;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/ScoStartEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
public class ScoStartEvent {
public boolean scoStart;
public ScoStartEvent(boolean scoStart){
this.scoStart = scoStart;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/SetFontSizeEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
import com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators.SmartGlassesFontSize;
public class SetFontSizeEvent {
public SmartGlassesFontSize fontSize;
public SetFontSizeEvent(SmartGlassesFontSize newFontSize){
this.fontSize = newFontSize;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/SmartGlassesConnectedEvent.java
================================================
package com.teamopensmartglasses.augmentoslib.events;
import com.teamopensmartglasses.smartglassesmanager.supportedglasses.SmartGlassesDevice;
public class SmartGlassesConnectedEvent {
public final SmartGlassesDevice device;
public SmartGlassesConnectedEvent(SmartGlassesDevice newDevice) {
this.device = newDevice;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/SmartGlassesConnectionEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
public class SmartGlassesConnectionEvent {
public final int connectionStatus;
public SmartGlassesConnectionEvent(int connectionStatus) {
this.connectionStatus = connectionStatus;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/TextToSpeechEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
public class TextToSpeechEvent {
public String text;
public String language; //the name of the language, fully spelled out, ll lowercase
public TextToSpeechEvent(String text, String language){
this.text = text;
this.language = language;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/eventbusmessages/VoskAudioChunkNewEvent.java
================================================
package com.teamopensmartglasses.smartglassesmanager.eventbusmessages;
public class VoskAudioChunkNewEvent {
public byte [] thisChunk;
public VoskAudioChunkNewEvent(byte [] thisChunk){
this.thisChunk = thisChunk;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/hci/AudioChunkCallback.java
================================================
package com.teamopensmartglasses.smartglassesmanager.hci;
import java.nio.ByteBuffer;
public interface AudioChunkCallback{
void onSuccess(ByteBuffer chunk);
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/hci/HearItBleMicrophone.java
================================================
package com.teamopensmartglasses.smartglassesmanager.hci;
import android.bluetooth.BluetoothAdapter;
import android.bluetooth.BluetoothDevice;
import android.bluetooth.BluetoothGatt;
import android.bluetooth.BluetoothGattCallback;
import android.bluetooth.BluetoothGattCharacteristic;
import android.bluetooth.BluetoothGattService;
import android.bluetooth.BluetoothProfile;
import android.content.Context;
import android.util.Log;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicBoolean;
public class HearItBleMicrophone {
private static final String TAG = "HearItBleMicrophone";
private static final UUID VOICE_SERVICE_UUID = UUID.fromString("00001853-0000-1000-8000-00805f9b34fb");
private static final UUID VOICE_DATA_CHARACTERISTIC_UUID = UUID.fromString("00002bcd-0000-1000-8000-00805f9b34fb");
private BluetoothAdapter bluetoothAdapter;
private BluetoothGatt bluetoothGatt;
private BluetoothGattCharacteristic voiceDataCharacteristic;
private Context context;
private AtomicBoolean isConnected = new AtomicBoolean(false);
private boolean foundHearIt = false;
private final int[] idxtbl = {-1, -1, -1, -1, 2, 4, 6, 8, -1, -1, -1, -1, 2, 4, 6, 8};
private final int[] steptbl = {
7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 23, 25, 28, 31, 34, 37, 41, 45,
50, 55, 60, 66, 73, 80, 88, 97, 107, 118, 130, 143, 157, 173, 190, 209, 230, 253, 279, 307,
337, 371, 408, 449, 494, 544, 598, 658, 724, 796, 876, 963, 1060, 1166, 1282, 1411, 1552, 1707, 1878, 2066,
2272, 2499, 2749, 3024, 3327, 3660, 4026, 4428, 4871, 5358, 5894, 6484, 7132, 7845, 8630, 9493, 10442, 11487, 12635, 13899,
15289, 16818, 18500, 20350, 22385, 24623, 27086, 29794, 32767
};
private int lastPacketNumber = -1;
public interface HearItBleMicCallback {
void onConnected();
void onPcmDataAvailable(byte[] pcmData);
}
private HearItBleMicCallback hearItBleMicCallback;
public void setHearItBleMicCallback(HearItBleMicCallback hearItBleMicCallback) {
this.hearItBleMicCallback = hearItBleMicCallback;
}
public boolean isConnected(){
return isConnected.get();
}
public HearItBleMicrophone(Context context) {
this.context = context;
this.bluetoothAdapter = BluetoothAdapter.getDefaultAdapter();
}
public void startScanning() {
if (!bluetoothAdapter.isEnabled()) {
Log.e(TAG, "Bluetooth is not enabled. Cannot start scanning.");
return;
}
bluetoothAdapter.startLeScan(leScanCallback);
}
private BluetoothAdapter.LeScanCallback leScanCallback = new BluetoothAdapter.LeScanCallback() {
@Override
public void onLeScan(BluetoothDevice device, int rssi, byte[] scanRecord) {
if (!foundHearIt && device.getName() != null && device.getName().startsWith("GMIC")) {
foundHearIt = true;
bluetoothAdapter.stopLeScan(leScanCallback);
connectToDevice(device);
}
}
};
private void connectToDevice(BluetoothDevice device) {
if (!bluetoothAdapter.isEnabled()) {
Log.e(TAG, "Bluetooth is not enabled. Cannot connect to device.");
return;
}
bluetoothGatt = device.connectGatt(context, false, gattCallback);
}
private final BluetoothGattCallback gattCallback = new BluetoothGattCallback() {
@Override
public void onConnectionStateChange(BluetoothGatt gatt, int status, int newState) {
if (newState == BluetoothProfile.STATE_CONNECTED) {
Log.i(TAG, "Connected to GATT server.");
bluetoothGatt.discoverServices();
} else if (newState == BluetoothProfile.STATE_DISCONNECTED) {
Log.i(TAG, "Disconnected from GATT server.");
isConnected.set(false);
}
}
@Override
public void onServicesDiscovered(BluetoothGatt gatt, int status) {
if (status == BluetoothGatt.GATT_SUCCESS) {
List services = bluetoothGatt.getServices();
for (BluetoothGattService service : services) {
if (VOICE_SERVICE_UUID.equals(service.getUuid())) {
voiceDataCharacteristic = service.getCharacteristic(VOICE_DATA_CHARACTERISTIC_UUID);
if (voiceDataCharacteristic != null) {
bluetoothGatt.setCharacteristicNotification(voiceDataCharacteristic, true);
isConnected.set(true);
if (hearItBleMicCallback != null) {
hearItBleMicCallback.onConnected();
}
}
}
}
}
}
@Override
public void onCharacteristicChanged(BluetoothGatt gatt, BluetoothGattCharacteristic characteristic) {
if (VOICE_DATA_CHARACTERISTIC_UUID.equals(characteristic.getUuid())) {
byte[] adpcmData = characteristic.getValue();
int sampleCount = adpcmData[3] * 2;
ByteBuffer pcmDataBuffer = ByteBuffer.allocate(sampleCount * 2).order(ByteOrder.LITTLE_ENDIAN); // Dynamically allocate buffer
adpcmToPcm(adpcmData, pcmDataBuffer);
byte[] pcmData = pcmDataBuffer.array();
if (hearItBleMicCallback != null) {
hearItBleMicCallback.onPcmDataAvailable(pcmData);
}
}
}
};
private void adpcmToPcm(byte[] adpcmData, ByteBuffer pcmDataBuffer) {
int sampleCount = adpcmData[3] * 2;
int currentPacketNumber = -1;
// Check and set currentPacketNumber
if (adpcmData.length == sampleCount / 2 + 6) {
currentPacketNumber = (adpcmData[adpcmData.length - 1] << 8) + adpcmData[adpcmData.length - 2];
if (lastPacketNumber != -1 && currentPacketNumber != lastPacketNumber + 1) {
int lostPackets = currentPacketNumber - lastPacketNumber - 1;
Log.w(TAG, "Warning: Packet loss detected, number of lost packets: " + lostPackets);
}
lastPacketNumber = currentPacketNumber;
}
int predict = (adpcmData[0] | (adpcmData[1] << 8)) << 16 >> 16;
int predict_idx = adpcmData[2];
byte[] pcode = new byte[adpcmData.length - 4];
System.arraycopy(adpcmData, 4, pcode, 0, pcode.length);
int code = pcode[0];
for (int i = 0; i < sampleCount; i++) {
int step = steptbl[predict_idx];
int diffq = step >> 3;
if ((code & 4) != 0) diffq += step;
step >>= 1;
if ((code & 2) != 0) diffq += step;
step >>= 1;
if ((code & 1) != 0) diffq += step;
if ((code & 8) != 0) predict -= diffq;
else predict += diffq;
predict = Math.max(-32768, Math.min(32767, predict));
predict_idx = Math.max(0, Math.min(88, predict_idx + idxtbl[code & 15]));
if (i % 2 == 0) code = pcode[i / 2];
else code >>= 4;
pcmDataBuffer.putShort((short) predict);
}
}
public void destroy() {
if (bluetoothGatt != null) {
bluetoothGatt.disconnect();
bluetoothGatt.close();
bluetoothGatt = null;
}
hearItBleMicCallback = null;
isConnected.set(false);
Log.i(TAG, "Bluetooth connection closed and callback turned off.");
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/hci/MicrophoneLocalAndBluetooth.java
================================================
package com.teamopensmartglasses.smartglassesmanager.hci;
//thanks to https://github.com/aahlenst/android-audiorecord-sample/blob/master/src/main/java/com/example/audiorecord/BluetoothRecordActivity.java
import android.bluetooth.BluetoothDevice;
import android.content.BroadcastReceiver;
import android.content.Context;
import android.content.Intent;
import android.content.IntentFilter;
import android.media.AudioFormat;
import android.media.AudioManager;
import android.media.AudioRecord;
import android.media.MediaRecorder;
import android.os.CountDownTimer;
import android.os.Handler;
import android.util.Log;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.ScoStartEvent;
import org.greenrobot.eventbus.EventBus;
import org.greenrobot.eventbus.Subscribe;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.concurrent.atomic.AtomicBoolean;
public class MicrophoneLocalAndBluetooth {
private static final String TAG = "WearableAi_MicrophoneLocalAndBluetooth";
private static final int SAMPLING_RATE_IN_HZ = 16000;
private static final int CHANNEL_CONFIG = AudioFormat.CHANNEL_IN_MONO;
private static final int AUDIO_FORMAT = AudioFormat.ENCODING_PCM_16BIT;
private final static float BUFFER_SIZE_SECONDS = 0.192f; // gives us 1024*3 = 3072 samples
private static final int BUFFER_SIZE_FACTOR = 2;
private final int bufferSize;
private boolean bluetoothAudio = false; // are we using local audio or bluetooth audio?
private boolean shouldUseHearItBleMicrophone = false; // should we use HearIt BLE microphone?
private int retries = 0;
private int retryLimit = 3;
private Handler mHandler;
private final AtomicBoolean recordingInProgress = new AtomicBoolean(false);
private HearItBleMicrophone hearItBleMicrophone;
private final BroadcastReceiver bluetoothStateReceiver = new BroadcastReceiver() {
private BluetoothState bluetoothState = BluetoothState.UNAVAILABLE;
@Override
public void onReceive(Context context, Intent intent) {
String action = intent.getAction();
if (action.equals(AudioManager.ACTION_SCO_AUDIO_STATE_UPDATED)) {
int state = intent.getIntExtra(AudioManager.EXTRA_SCO_AUDIO_STATE, -1);
switch (state) {
case AudioManager.SCO_AUDIO_STATE_CONNECTED:
if (mIsStarting) {
mIsStarting = false;
}
if (mIsCountDownOn) {
mIsCountDownOn = false;
mCountDown.cancel();
}
bluetoothAudio = true;
startRecording();
break;
case AudioManager.SCO_AUDIO_STATE_CONNECTING:
handleBluetoothStateChange(BluetoothState.UNAVAILABLE);
case AudioManager.SCO_AUDIO_STATE_DISCONNECTED:
handleBluetoothStateChange(BluetoothState.UNAVAILABLE);
break;
case AudioManager.SCO_AUDIO_STATE_ERROR:
handleBluetoothStateChange(BluetoothState.UNAVAILABLE);
break;
}
} else if (action.equals(BluetoothDevice.ACTION_ACL_CONNECTED)) {
handleNewBluetoothDevice();
} else if (action.equals(BluetoothDevice.ACTION_ACL_DISCONNECTED)) {
handleDisconnectBluetoothDevice();
}
}
private void handleBluetoothStateChange(BluetoothState state) {
if (bluetoothState == state) {
return;
}
bluetoothState = state;
bluetoothStateChanged(state);
}
private void handleNewBluetoothDevice() {
retries = 0;
mIsCountDownOn = true;
mCountDown.start();
}
private void handleDisconnectBluetoothDevice() {
if (mIsCountDownOn) {
mIsCountDownOn = false;
mCountDown.cancel();
}
bluetoothAudio = false;
deactivateBluetoothSco();
startRecording();
}
};
private AudioRecord recorder = null;
private AudioManager audioManager;
private boolean mIsCountDownOn = false;
private boolean mIsStarting = false;
private Thread recordingThread = null;
private Context mContext;
private AudioChunkCallback mChunkCallback;
public MicrophoneLocalAndBluetooth(Context context, boolean useBluetoothSco, AudioChunkCallback chunkCallback) {
this(context, chunkCallback);
this.shouldUseHearItBleMicrophone = true;
useBluetoothMic(useBluetoothSco);
}
public MicrophoneLocalAndBluetooth(Context context, AudioChunkCallback chunkCallback) {
bufferSize = Math.round(SAMPLING_RATE_IN_HZ * BUFFER_SIZE_SECONDS);
mIsStarting = true;
mContext = context;
audioManager = (AudioManager) mContext.getSystemService(Context.AUDIO_SERVICE);
mChunkCallback = chunkCallback;
mHandler = new Handler();
startRecording();
}
private void useBluetoothMic(boolean shouldUseBluetoothSco) {
bluetoothAudio = shouldUseBluetoothSco;
if (shouldUseBluetoothSco) {
startBluetoothSco();
} else {
stopBluetoothSco();
}
if (recordingInProgress.get()) {
startRecording();
}
}
private void startBluetoothSco() {
mContext.registerReceiver(bluetoothStateReceiver, new IntentFilter(
AudioManager.ACTION_SCO_AUDIO_STATE_UPDATED));
mContext.registerReceiver(bluetoothStateReceiver,
new IntentFilter(BluetoothDevice.ACTION_ACL_CONNECTED));
mContext.registerReceiver(bluetoothStateReceiver,
new IntentFilter(BluetoothDevice.ACTION_ACL_DISCONNECTED));
mIsCountDownOn = true;
mCountDown.start();
}
private void stopBluetoothSco() {
mIsCountDownOn = false;
mCountDown.cancel();
try {
mContext.unregisterReceiver(bluetoothStateReceiver);
} catch (IllegalArgumentException e) {
e.printStackTrace();
}
}
private void startRecording() {
if (recorder != null) {
stopRecording();
}
if (bluetoothAudio) {
audioManager.setMode(AudioManager.MODE_IN_CALL);
EventBus.getDefault().post(new ScoStartEvent(true));
} else {
audioManager.setMode(AudioManager.MODE_NORMAL);
EventBus.getDefault().post(new ScoStartEvent(false));
}
recorder = new AudioRecord(MediaRecorder.AudioSource.UNPROCESSED,
SAMPLING_RATE_IN_HZ, CHANNEL_CONFIG, AUDIO_FORMAT, bufferSize * 2);
recorder.startRecording();
recordingInProgress.set(true);
recordingThread = new Thread(new RecordingRunnable(), "Recording Thread");
recordingThread.start();
if (bluetoothAudio && shouldUseHearItBleMicrophone) {
Log.d(TAG, "Connecting HearItBle... ");
hearItBleMicrophone = new HearItBleMicrophone(mContext);
hearItBleMicrophone.setHearItBleMicCallback(new HearItBleMicrophone.HearItBleMicCallback() {
@Override
public void onConnected() {
Log.d(TAG, "--- HearItBle connected!");
stopAndroidMics();
}
@Override
public void onPcmDataAvailable(byte[] pcmData) {
ByteBuffer b_buffer = ByteBuffer.allocate(pcmData.length);
b_buffer.put(pcmData);
mChunkCallback.onSuccess(b_buffer);
}
});
hearItBleMicrophone.startScanning();
}
}
private void stopAndroidMics(){
mIsCountDownOn = false;
mCountDown.cancel();
deactivateBluetoothSco();
audioManager.setMode(AudioManager.MODE_NORMAL);
stopRecording();
}
private void stopRecording() {
Log.d(TAG, "Running stopRecording...");
if (recorder == null) {
Log.d(TAG, "--- Recorder null, exiting.");
return;
}
recordingInProgress.set(false);
recorder.stop();
recorder.release();
recorder = null;
recordingThread = null;
}
private void activateBluetoothSco() {
retries += 1;
if (!audioManager.isBluetoothScoAvailableOffCall()) {
Log.e(TAG, "SCO is not available, recording is not possible");
return;
}
if (audioManager.isBluetoothScoOn()) {
audioManager.stopBluetoothSco();
}
audioManager.startBluetoothSco();
}
private void deactivateBluetoothSco() {
audioManager.stopBluetoothSco();
}
private void bluetoothStateChanged(BluetoothState state) {
if (BluetoothState.UNAVAILABLE == state && recordingInProgress.get()) {
bluetoothAudio = false;
stopRecording();
deactivateBluetoothSco();
} else if (BluetoothState.AVAILABLE == state && !recordingInProgress.get()) {
bluetoothAudio = true;
startRecording();
} else if (BluetoothState.AVAILABLE == state && !bluetoothAudio) {
bluetoothAudio = true;
startRecording();
}
}
private class RecordingRunnable implements Runnable {
@Override
public void run() {
short[] short_buffer = new short[bufferSize];
ByteBuffer b_buffer = ByteBuffer.allocate(short_buffer.length * 2);
while (recordingInProgress.get()) {
int result = recorder.read(short_buffer, 0, short_buffer.length);
if (result < 0) {
Log.d(TAG, "ERROR");
}
b_buffer.order(ByteOrder.LITTLE_ENDIAN);
b_buffer.asShortBuffer().put(short_buffer);
if (hearItBleMicrophone != null && !hearItBleMicrophone.isConnected()) {
mChunkCallback.onSuccess(b_buffer);
}
b_buffer.clear();
}
}
private String getBufferReadFailureReason(int errorCode) {
switch (errorCode) {
case AudioRecord.ERROR_INVALID_OPERATION:
return "ERROR_INVALID_OPERATION";
case AudioRecord.ERROR_BAD_VALUE:
return "ERROR_BAD_VALUE";
case AudioRecord.ERROR_DEAD_OBJECT:
return "ERROR_DEAD_OBJECT";
case AudioRecord.ERROR:
return "ERROR";
default:
return "Unknown (" + errorCode + ")";
}
}
}
enum BluetoothState {
AVAILABLE, UNAVAILABLE
}
private CountDownTimer mCountDown = new CountDownTimer(1201, 400) {
@SuppressWarnings("synthetic-access")
@Override
public void onTick(long millisUntilFinished) {
audioManager.startBluetoothSco();
}
@SuppressWarnings("synthetic-access")
@Override
public void onFinish() {
mIsCountDownOn = false;
bluetoothAudio = false;
startRecording();
}
};
public void destroy() {
stopRecording();
if (hearItBleMicrophone != null) {
hearItBleMicrophone.destroy();
}
mIsCountDownOn = false;
mCountDown.cancel();
deactivateBluetoothSco();
audioManager.setMode(AudioManager.MODE_NORMAL);
if (mContext != null) {
try {
mContext.unregisterReceiver(bluetoothStateReceiver);
} catch (IllegalArgumentException e) {
e.printStackTrace();
}
}
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/smartglassescommunicators/AndroidSGC.java
================================================
package com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators;
import android.content.Context;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.os.Handler;
import android.os.HandlerThread;
import android.os.StrictMode;
import android.util.Base64;
import android.util.Log;
import com.teamopensmartglasses.smartglassesmanager.comms.AspWebsocketServer;
import com.teamopensmartglasses.smartglassesmanager.comms.AudioSystem;
import com.teamopensmartglasses.smartglassesmanager.comms.MessageTypes;
import com.teamopensmartglasses.augmentoslib.events.GlassesPovImageEvent;
import com.teamopensmartglasses.smartglassesmanager.utils.NetworkUtils;
import org.greenrobot.eventbus.EventBus;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.net.DatagramSocket;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.Random;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.TimeUnit;
import io.reactivex.rxjava3.subjects.PublishSubject;
public class AndroidSGC extends SmartGlassesCommunicator {
private static final String TAG = "WearableAi_AndroidSGC";
PublishSubject dataObservable;
private static boolean killme;
private static Handler heart_beat_handler;
//handler for advertising
private Handler adv_handler;
//network details
public int PORT_NUM = 8891;
public DatagramSocket adv_socket;
public String adv_key = "WearableAiCyborg";
//SOCKET STUFF
//socket
public AspWebsocketServer asgWebSocket;
public void displayRowsCard(String[] rowStrings){
}
//acutal socket
ServerSocket serverSocket;
Socket socket;
boolean shouldDie = false;
//socket threads
Thread SocketThread = null;
Thread ReceiveThread = null;
Thread SendThread = null;
//queue of data to send through the socket
public BlockingQueue queue;
//address info
public final int SERVER_PORT = 4567;
//i/o
public DataOutputStream output;
public DataInputStream input;
public int outbound_heart_beats = 0;
//other
final byte [] ack_id = {0x13, 0x37};
final byte [] heart_beat_id = {0x19, 0x20};
final byte [] img_id = {0x01, 0x10}; //id for images
//audio streaming system
AudioSystem audioSystem;
Context context;
public AndroidSGC(Context context, PublishSubject dataObservable){
super();
this.dataObservable = dataObservable;
this.context = context;
//create a new queue to hold outbound message
queue = new ArrayBlockingQueue(50);
killme = false;
//state information
mConnectState = 0;
}
//not used/valid yet
@Override
protected void setFontSizes(){
LARGE_FONT = 3;
MEDIUM_FONT = 2;
SMALL_FONT = 0;
}
public void connectToSmartGlasses(){
//open the UDP socket to broadcast our IP address
openSocket();
//send broadcast over UDP that tells smart glasses they can find us
adv_handler = new Handler();
final int delay = 1000; // 1000 milliseconds == 1 second
adv_handler.postDelayed(new Runnable() {
public void run() {
new Thread(new SendAdvThread()).start();
adv_handler.postDelayed(this, delay);
}
}, 5);
startAsgWebSocketConnection();
audioSystem = new AudioSystem(context, dataObservable);
//start first socketThread
Log.d(TAG, "running start socket");
startSocket();
}
class SendAdvThread extends Thread {
public void run() {
//send broadcast so smart glasses know our address
NetworkUtils.sendBroadcast(adv_key, adv_socket, PORT_NUM, context);
}
}
public void blankScreen(){
// try{
// //build json object to send command result
// JSONObject commandResponseObject = new JSONObject();
// commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.ACTION_SWITCH_MODES);
// commandResponseObject.put(MessageTypes.NEW_MODE, MessageTypes.MODE_BLANK);
//
// //send the command result to web socket, to send to asg
// dataObservable.onNext(commandResponseObject);
// } catch (JSONException e){
// e.printStackTrace();
// }
}
public void startAsgWebSocketConnection(){
Log.d(TAG, "Starting WebSocket Server");
//String address = "localhost:8887";
//InetSocketAddress inetSockAddress = new InetSocketAddress(address);
int port = 8887;
asgWebSocket = new AspWebsocketServer(port);
asgWebSocket.setObservable(dataObservable);
asgWebSocket.start();
Log.d(TAG, "WebSocket Server STARTED");
}
//SOCKET STUFF
public void startSocket(){
//start first socketThread
Log.d(TAG, "socket val in startSocket: " + socket);
if (socket == null) {
Log.d(TAG, "starting new SocketThread" + socket);
connectionEvent(1);
SocketThread = new Thread(new SocketThread());
SocketThread.start();
//setup handler to handle keeping connection alive, all subsequent start of SocketThread
//start a new handler thread to send heartbeats
HandlerThread thread = new HandlerThread("HeartBeater");
thread.start();
heart_beat_handler = new Handler(thread.getLooper());
final int hb_delay = 3000;
final int min_hb_delay = 1000;
final int max_hb_delay = 2000;
Random rand = new Random();
heart_beat_handler.postDelayed(new Runnable() {
public void run() {
heartBeat();
//random hb_delay for heart beat so as to disallow synchronized failure between client and server
int random_hb_delay = rand.nextInt((max_hb_delay - min_hb_delay) + 1) + min_hb_delay;
heart_beat_handler.postDelayed(this, random_hb_delay);
}
}, hb_delay);
} else {
Log.d(TAG, "socket wasn't null, so not starting");
}
}
public void openSocket() {
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(policy);
try {
//Open a random port to send the package
adv_socket = new DatagramSocket();
adv_socket.setBroadcast(true);
} catch (IOException e) {
Log.e(TAG, "IOException: " + e.getMessage());
}
}
class SocketThread implements Runnable {
@Override
public void run() {
Log.d(TAG, "I have started SOCKETTHREAD");
try {
if (killme){
Log.d(TAG, "I have killed myself");
return;
}
Log.d(TAG, "Starting new socket, waiting for connection...");
serverSocket = new ServerSocket(SERVER_PORT);
//serverSocket.setSoTimeout(2000);
try {
socket = serverSocket.accept();
if (killme){
return;
}
socket.setSoTimeout(5000);
Log.d(TAG, "Got socket connection.");
//output = new PrintWriter(socket.getOutputStream(), true);
output = new DataOutputStream(socket.getOutputStream());
input = new DataInputStream(new DataInputStream(socket.getInputStream()));
connectionEvent(2);
if (ReceiveThread == null) { //if the thread is null, make a new one (the first one)
ReceiveThread = new Thread(new ReceiveThread());
ReceiveThread.start();
} else if (!ReceiveThread.isAlive()) { //if the thread is not null but it's dead, let it join then start a new one
try {
ReceiveThread.join(); //make sure socket thread has joined before throwing off a new one
} catch (InterruptedException e) {
e.printStackTrace();
}
ReceiveThread = new Thread(new ReceiveThread());
ReceiveThread.start();
}
if (SendThread == null) { //if the thread is null, make a new one (the first one)
SendThread = new Thread(new SendThread());
SendThread.start();
} else if (!SendThread.isAlive()) { //if the thread is not null but it's dead, let it join then start a new one
try {
SendThread.join(); //make sure socket thread has joined before throwing off a new one
} catch (InterruptedException e) {
e.printStackTrace();
}
SendThread = new Thread(new SendThread());
SendThread.start();
}
} catch (IOException e) {
e.printStackTrace();
connectionEvent(0);
}
} catch (IOException e) {
e.printStackTrace();
connectionEvent(0);
}
}
}
//receives messages
public void heartBeat(){
//check if we are still connected.
//if not , reconnect,
//if we are connected, send a heart beat to make sure we are still connected
if (mConnectState == 0 && !killme) {
Log.d(TAG, "heartBeat failing");
restartSocket();
} else if (mConnectState == 2){
//make sure we don't have a ton of outbound heart beats unresponded to
if (outbound_heart_beats > 5) {
Log.d(TAG, "heartBeat outbounds failing");
restartSocket();
return;
}
//increment counter
outbound_heart_beats++;
//send heart beat
sendBytes(heart_beat_id, null);
}
}
//receives messages
public class ReceiveThread implements Runnable {
@Override
public void run() {
//System.out.println("Receive Started, mconnect: " + mConnectState);
while (true) {
if (killme){
return;
}
if (mConnectState != 2){
break;
}
byte b1, b2;
byte [] raw_data = null;
byte goodbye1, goodbye2, goodbye3;
try {
byte hello1 = input.readByte(); // read hello of incoming message
byte hello2 = input.readByte(); // read hello of incoming message
byte hello3 = input.readByte(); // read hello of incoming message
//make sure header is verified
if (hello1 != 0x01 || hello2 != 0x02 || hello3 != 0x03){
Log.d(TAG, "Receive thread: broken intro fail");
break;
}
//length of body
int body_len = input.readInt();
//read in message id bytes
b1 = input.readByte();
b2 = input.readByte();
//read in message body (if there is one)
if (body_len > 0){
raw_data = new byte[body_len];
input.readFully(raw_data, 0, body_len); // read the body
}
goodbye1 = input.readByte(); // read goodbye of incoming message
goodbye2 = input.readByte(); // read goodbye of incoming message
goodbye3 = input.readByte(); // read goodbye of incoming message
} catch (IOException e) {
if (killme){
Log.d(TAG, "Socket closed (by us), cleaning up.");
} else {
Log.d(TAG, "Socket closed.");
e.printStackTrace();
}
break;
}
//make sure footer is verified
if (goodbye1 != 0x03 || goodbye2 != 0x02 || goodbye3 != 0x01) {
Log.d(TAG, "Receive thread: broken footer fail");
break;
}
//now process the data that was sent to us
if ((b1 == heart_beat_id[0]) && (b2 == heart_beat_id[1])){ //heart beat id tag
outbound_heart_beats--;
} else if ((b1 == ack_id[0]) && (b2 == ack_id[1])){ //an ack id
} else if ((b1 == img_id[0]) && (b2 == img_id[1])){ //an ack id
Log.d(TAG, "Got IMAGE");
if (raw_data != null) {
//remember the time we received it
long imageTime = System.currentTimeMillis();
//ping back the client to let it know we received the message
sendBytes(ack_id, null);
handleImage(raw_data, imageTime);
}
} else {
Log.d(TAG, "Killing cuz corrupted data");
break;
}
}
Log.d(TAG, "Receive thread throw broken socket");
throwBrokenSocket();
}
}
public void restartSocket(){
Log.d(TAG, "Restarting socket.");
connectionEvent(1);
outbound_heart_beats = 0;
//close the previous socket now that it's broken/being restarted
killSocket();
//make sure socket thread has joined before throwing off a new one
try {
SocketThread.join();
} catch (InterruptedException e){
e.printStackTrace();
}
//start a new socket thread
SocketThread = new Thread(new SocketThread());
SocketThread.start();
}
public void killSocket(){
try {
Log.d(TAG, "Closing socket, input, serverSocket, etc.");
if (serverSocket != null && (!serverSocket.isClosed())) {
serverSocket.close();
serverSocket = null;
}
if (socket != null){
socket.close();
socket = null;
}
if (output != null){
output.close();
output = null;
}
if (input != null){
input.close();
input = null;
}
} catch (IOException e) {
Log.d(TAG, "killSocket failed");
e.printStackTrace();
}
}
public byte[] my_int_to_bb_be(int myInteger){
return ByteBuffer.allocate(4).order(ByteOrder.BIG_ENDIAN).putInt(myInteger).array();
}
public void sendBytes(byte[] id, byte [] data){
//first, send hello
byte [] hello = {0x01, 0x02, 0x03};
//then send length of body
byte[] len;
if (data != null) {
len = my_int_to_bb_be(data.length);
} else {
len = my_int_to_bb_be(0);
}
//then send id of message type
byte [] msg_id = id;
//then send data
byte [] body = data;
//then send end tag - eventually make this unique to the image
byte [] goodbye = {0x3, 0x2, 0x1};
//combine those into a payload
ByteArrayOutputStream outputStream;
try {
outputStream = new ByteArrayOutputStream();
outputStream.write(hello);
outputStream.write(len);
outputStream.write(msg_id);
if (body != null) {
outputStream.write(body);
}
outputStream.write(goodbye);
} catch (IOException e){
connectionEvent(0);
return;
}
byte [] payload = outputStream.toByteArray();
//send it in a background thread
//new Thread(new SendThread(payload)).start();
queue.add(payload);
}
//this sends messages
class SendThread implements Runnable {
SendThread() {
}
@Override
public void run() {
queue.clear();
while (true){
if (killme){
return;
}
if (mConnectState != 2){
break;
}
if (queue.size() > 10){
break;
}
byte [] data;
try {
data = queue.poll(100, TimeUnit.MILLISECONDS); //block until there is something we can pull out to send
if (data == null){
continue;
}
} catch (InterruptedException e){
e.printStackTrace();
break;
}
try {
output.write(data); // write the message
} catch (java.io.IOException e) {
e.printStackTrace();
break;
}
}
Log.d(TAG, "Send thread throw broken socket");
throwBrokenSocket();
}
}
public void throwBrokenSocket(){
if (mConnectState == 2){
Log.d(TAG, "Throwing broken socket");
connectionEvent(0);
}
}
//^^^ SOCKET STUFF
public int getConnectionState(){
return mConnectState;
}
public void destroy(){
Log.d(TAG, "Destroying AndroidSGC");
killme = true;
//kill AudioSystem
audioSystem.destroy();
//kill asgWebSocket
asgWebSocket.destroy();
//stop sending heart beats
heart_beat_handler.removeCallbacksAndMessages(null);
//stop advertising broadcasting IP
if (adv_handler != null) {
adv_handler.removeCallbacksAndMessages(null);
}
//stop sockets
killSocket();
//kill this socket
try {
Log.i(TAG, "SOCKETTHREAD TRYNA JOIN");
if (SocketThread != null) {
SocketThread.join();
}
Log.i(TAG, "SOCKETTHREAD JOINED");
Log.i(TAG, "SENDTTHREAD TRYNA JOIN");
if (SendThread != null) {
SendThread.join();
}
Log.i(TAG, "SENDTTHREAD JOINED");
Log.i(TAG, "RECEIVE THREAD TRYNA JOIN");
if (ReceiveThread != null) {
ReceiveThread.join();
}
Log.i(TAG, "RECEIVE THREAD JOINED");
} catch (InterruptedException e){
e.printStackTrace();
Log.d(TAG, "Error waiting for threads to joing");
}
}
public void displayReferenceCardSimple(String title, String body){
try{
//build json object to send command result
JSONObject commandResponseObject = new JSONObject();
commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.REFERENCE_CARD_SIMPLE_VIEW);
commandResponseObject.put(MessageTypes.REFERENCE_CARD_SIMPLE_VIEW_TITLE, title);
commandResponseObject.put(MessageTypes.REFERENCE_CARD_SIMPLE_VIEW_BODY, body);
//send the command result to web socket, to send to asg
dataObservable.onNext(commandResponseObject);
} catch (JSONException e){
e.printStackTrace();
}
}
public void displayTextWall(String text){
Log.d(TAG, "SHOWING TEXT WALL");
try{
//build json object to send command result
JSONObject commandResponseObject = new JSONObject();
commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.REFERENCE_CARD_TEXT_WALL_VIEW);
commandResponseObject.put(MessageTypes.REFERENCE_CARD_TEXT_WALL_TEXT, text);
//send the command result to web socket, to send to asg
dataObservable.onNext(commandResponseObject);
} catch (JSONException e){
e.printStackTrace();
}
}
public void displayDoubleTextWall(String textTop, String textBottom){
Log.d(TAG, "TODO: SHOWING DOUBLE TEXT WALL");
displayTextWall(textTop + textBottom);
}
public void displayReferenceCardImage(String title, String body, String imgUrl){
try{
//build json object to send command result
JSONObject commandResponseObject = new JSONObject();
commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.REFERENCE_CARD_IMAGE_VIEW);
commandResponseObject.put(MessageTypes.REFERENCE_CARD_IMAGE_VIEW_TITLE, title);
commandResponseObject.put(MessageTypes.REFERENCE_CARD_IMAGE_VIEW_BODY, body);
commandResponseObject.put(MessageTypes.REFERENCE_CARD_IMAGE_VIEW_IMG_URL, imgUrl);
//send the command result to web socket, to send to asg
dataObservable.onNext(commandResponseObject);
} catch (JSONException e){
e.printStackTrace();
}
}
@Override
public void displayBitmap(Bitmap bmp) {
}
public void displayBulletList(String title, String [] bullets){
displayBulletList(title, bullets, 0);
}
public void displayBulletList(String title, String [] bullets, int lingerTime){
}
public void stopScrollingTextViewMode() {
Log.d(TAG, "STOP SCROLLING TEXT VIEW");
try{
//build json object to send command result
JSONObject commandResponseObject = new JSONObject();
commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.SCROLLING_TEXT_VIEW_STOP);
//send the command result to web socket, to send to asg
dataObservable.onNext(commandResponseObject);
} catch (JSONException e){
e.printStackTrace();
}
}
public void startScrollingTextViewMode(String title){
super.startScrollingTextViewMode(title);
Log.d(TAG, "START SCROLLING TEXT VIEW");
try{
//build json object to send command result
JSONObject commandResponseObject = new JSONObject();
commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.SCROLLING_TEXT_VIEW_START);
commandResponseObject.put(MessageTypes.SCROLLING_TEXT_VIEW_TITLE, title);
//send the command result to web socket, to send to asg
dataObservable.onNext(commandResponseObject);
} catch (JSONException e){
e.printStackTrace();
}
}
public void scrollingTextViewIntermediateText(String text){
try{
//build json object to send command result
JSONObject commandResponseObject = new JSONObject();
commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.SCROLLING_TEXT_VIEW_INTERMEDIATE);
commandResponseObject.put(MessageTypes.SCROLLING_TEXT_VIEW_TEXT, text);
//send the command result to web socket, to send to asg
dataObservable.onNext(commandResponseObject);
} catch (JSONException e){
e.printStackTrace();
}
}
public void scrollingTextViewFinalText(String text){
try{
//build json object to send command result
JSONObject commandResponseObject = new JSONObject();
commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.SCROLLING_TEXT_VIEW_FINAL);
commandResponseObject.put(MessageTypes.SCROLLING_TEXT_VIEW_TEXT, text);
//send the command result to web socket, to send to asg
dataObservable.onNext(commandResponseObject);
} catch (JSONException e){
e.printStackTrace();
}
}
public void showHomeScreen(){
try{
//build json object to send command result
JSONObject commandResponseObject = new JSONObject();
commandResponseObject.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.ACTION_SWITCH_MODES);
commandResponseObject.put(MessageTypes.NEW_MODE, MessageTypes.MODE_HOME);
//send the command result to web socket, to send to asg
dataObservable.onNext(commandResponseObject);
} catch (JSONException e){
e.printStackTrace();
}
}
public void displayPromptView(String prompt, String [] options){
//generate args list
if (options != null) {
//required args
// try{
// JSONArray argsList = new JSONArray();
// for (String s : options) {
// argsList.put(s);
// }
// JSONObject wakeWordFoundEvent = new JSONObject();
// wakeWordFoundEvent.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.VOICE_COMMAND_STREAM_EVENT);
// wakeWordFoundEvent.put(MessageTypes.VOICE_COMMAND_STREAM_EVENT_TYPE, MessageTypes.REQUIRED_ARG_EVENT_TYPE);
// wakeWordFoundEvent.put(MessageTypes.ARG_NAME, prompt);
// wakeWordFoundEvent.put(MessageTypes.ARG_OPTIONS, argsList);
// dataObservable.onNext(wakeWordFoundEvent);
// } catch (JSONException e){
// e.printStackTrace();
// }
//natural language arg
// try {
// JSONObject commandFoundEvent = new JSONObject();
// commandFoundEvent.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.VOICE_COMMAND_STREAM_EVENT);
// commandFoundEvent.put(MessageTypes.VOICE_COMMAND_STREAM_EVENT_TYPE, MessageTypes.COMMAND_EVENT_TYPE);
// commandFoundEvent.put(MessageTypes.INPUT_VOICE_COMMAND_NAME, command);
// commandFoundEvent.put(MessageTypes.INPUT_WAKE_WORD, this.wakeWordGiven);
// commandFoundEvent.put(MessageTypes.VOICE_ARG_EXPECT_TYPE, MessageTypes.VOICE_ARG_EXPECT_NATURAL_LANGUAGE);
// dataObservable.onNext(commandFoundEvent);
// } catch (JSONException e){
// e.printStackTrace();
// }
//found wake word
JSONArray argsList = new JSONArray();
for (String s : options) {
argsList.put(s);
}
try {
JSONObject wakeWordFoundEvent = new JSONObject();
wakeWordFoundEvent.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.VOICE_COMMAND_STREAM_EVENT);
wakeWordFoundEvent.put(MessageTypes.VOICE_COMMAND_STREAM_EVENT_TYPE, MessageTypes.WAKE_WORD_EVENT_TYPE);
wakeWordFoundEvent.put(MessageTypes.VOICE_COMMAND_LIST, argsList.toString());
wakeWordFoundEvent.put(MessageTypes.INPUT_WAKE_WORD, prompt);
dataObservable.onNext(wakeWordFoundEvent);
} catch (JSONException e){
e.printStackTrace();
}
}
}
public void displayTextLine(String text){
displayReferenceCardSimple("", text);
}
public void displayCenteredText(String text){
//TODO: Complete this
displayTextLine(text);
}
public void displayCustomContent(String json) {
displayReferenceCardSimple("CustomDisplayNotImplemented", json);
}
public void showNaturalLanguageCommandScreen(String prompt, String naturalLanguageArgs){
try {
JSONObject commandFoundEvent = new JSONObject();
commandFoundEvent.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.VOICE_COMMAND_STREAM_EVENT);
commandFoundEvent.put(MessageTypes.VOICE_COMMAND_STREAM_EVENT_TYPE, MessageTypes.COMMAND_EVENT_TYPE);
commandFoundEvent.put(MessageTypes.INPUT_VOICE_COMMAND_NAME, "myCommand");
commandFoundEvent.put(MessageTypes.INPUT_WAKE_WORD, "myWakeWord");
commandFoundEvent.put(MessageTypes.VOICE_ARG_EXPECT_TYPE, MessageTypes.VOICE_ARG_EXPECT_NATURAL_LANGUAGE);
dataObservable.onNext(commandFoundEvent);
} catch (JSONException e){
e.printStackTrace();
}
}
public void updateNaturalLanguageCommandScreen(String naturalLanguageArgs){
try {
JSONObject commandFoundEvent = new JSONObject();
commandFoundEvent.put(MessageTypes.MESSAGE_TYPE_LOCAL, MessageTypes.VOICE_COMMAND_STREAM_EVENT);
commandFoundEvent.put(MessageTypes.VOICE_COMMAND_STREAM_EVENT_TYPE, MessageTypes.COMMAND_ARGS_EVENT_TYPE);
commandFoundEvent.put(MessageTypes.INPUT_VOICE_STRING, naturalLanguageArgs);
dataObservable.onNext(commandFoundEvent);
} catch (JSONException e) {
e.printStackTrace();
}
}
public void handleImage(byte [] raw_data, long imageTime){
//convert to bitmap
Bitmap bitmap = BitmapFactory.decodeByteArray(raw_data, 0, raw_data.length);
//save and process 1 image at set frequency
sendPovImage(raw_data, imageTime);
}
public void sendPovImage(byte [] img, long imageTime){
String encodedImage = Base64.encodeToString(img, Base64.DEFAULT);
EventBus.getDefault().post(new GlassesPovImageEvent(encodedImage, imageTime));
}
public void setFontSize(SmartGlassesFontSize fontSize){}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/smartglassescommunicators/AudioWearableSGC.java
================================================
package com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators;
import android.content.Context;
import android.graphics.Bitmap;
import android.util.Log;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.TextToSpeechEvent;
import org.greenrobot.eventbus.EventBus;
public class AudioWearableSGC extends SmartGlassesCommunicator {
private static final String TAG = "WearableAi_AndroidWearableSGC";
private static boolean killme;
Context context;
public AudioWearableSGC(Context context){
super();
//state information
killme = false;
mConnectState = 0;
}
public void setFontSizes(){
}
public void connectToSmartGlasses(){
connectionEvent(2);
}
public void blankScreen(){
}
public void displayRowsCard(String[] rowStrings){
}
public void destroy(){
killme = true;
}
public void displayReferenceCardSimple(String title, String body){
Log.d(TAG, "TTS reference card");
EventBus.getDefault().post(new TextToSpeechEvent(title + ", " + body, "english"));
}
public void displayReferenceCardImage(String title, String body, String imgUrl){
Log.d(TAG, "TTS reference card");
EventBus.getDefault().post(new TextToSpeechEvent(title + ", " + body, "english"));
}
public void displayBulletList(String title, String [] bullets){
displayBulletList(title, bullets, 0);
}
public void displayBulletList(String title, String [] bullets, int lingerTime){
}
public void displayTextWall(String text){}
public void displayDoubleTextWall(String textTop, String textBottom){}
public void stopScrollingTextViewMode() {
}
public void startScrollingTextViewMode(String title){
}
public void scrollingTextViewIntermediateText(String text){
}
public void scrollingTextViewFinalText(String text){
}
public void showHomeScreen(){
}
public void displayPromptView(String prompt, String [] options){
}
public void displayTextLine(String text){
Log.d(TAG, "displayTextLine: " + text);
EventBus.getDefault().post(new TextToSpeechEvent(text, "english"));
}
@Override
public void displayBitmap(Bitmap bmp) {
}
public void displayCenteredText(String text){
//TODO: Find a way to add (optional) pauses between lines?
displayTextLine(text);
}
@Override
public void displayCustomContent(String json) {
displayTextLine(json);
}
public void showNaturalLanguageCommandScreen(String prompt, String naturalLanguageArgs){
}
public void updateNaturalLanguageCommandScreen(String naturalLanguageArgs){
}
public void setFontSize(SmartGlassesFontSize fontSize){}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/smartglassescommunicators/SmartGlassesCommunicator.java
================================================
package com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators;
import android.graphics.Bitmap;
import com.teamopensmartglasses.augmentoslib.events.GlassesTapOutputEvent;
import com.teamopensmartglasses.smartglassesmanager.eventbusmessages.SmartGlassesConnectionEvent;
import org.greenrobot.eventbus.EventBus;
public abstract class SmartGlassesCommunicator {
//basic glasses utils/settings
public int mConnectState = 0;
protected SmartGlassesModes currentMode;
public abstract void connectToSmartGlasses();
public abstract void blankScreen();
public abstract void destroy();
public final String commandNaturalLanguageString = "Command: ";
public final String finishNaturalLanguageString = "'finish command' when done";
//reference card
public abstract void displayReferenceCardSimple(String title, String body);
//display text wall
public abstract void displayTextWall(String text);
public abstract void displayDoubleTextWall(String textTop, String textBottom);
public abstract void displayReferenceCardImage(String title, String body, String imgUrl);
public abstract void displayBulletList(String title, String [] bullets);
public abstract void displayRowsCard(String[] rowStrings);
//voice command UI
public abstract void showNaturalLanguageCommandScreen(String prompt, String naturalLanguageArgs);
public abstract void updateNaturalLanguageCommandScreen(String naturalLanguageArgs);
//scrolling text view
public void startScrollingTextViewMode(String title){
setMode(SmartGlassesModes.SCROLLING_TEXT_VIEW);
}
public abstract void scrollingTextViewIntermediateText(String text);
public abstract void scrollingTextViewFinalText(String text);
public abstract void stopScrollingTextViewMode();
//prompt view card
public abstract void displayPromptView(String title, String [] options);
//display text line
public abstract void displayTextLine(String text);
public abstract void displayBitmap(Bitmap bmp);
//display centered text
public abstract void displayCenteredText(String text);
public abstract void displayCustomContent(String json);
//home screen
public abstract void showHomeScreen();
public abstract void setFontSize(SmartGlassesFontSize fontSize);
//fonts
public int LARGE_FONT;
public int MEDIUM_FONT;
public int SMALL_FONT;
public SmartGlassesCommunicator(){
setFontSizes();
}
//must be run and set font sizes
protected abstract void setFontSizes();
public int getConnectionState(){
return mConnectState;
}
protected boolean isConnected(){
return (mConnectState == 2);
}
public void connectionEvent(int connectState){
mConnectState = connectState;
EventBus.getDefault().post(new SmartGlassesConnectionEvent(mConnectState));
if (isConnected()) {
showHomeScreen();
}
}
public void tapEvent(int num){
EventBus.getDefault().post(new GlassesTapOutputEvent(num, false, System.currentTimeMillis()));
}
public void setMode(SmartGlassesModes mode){
currentMode = mode;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/smartglassescommunicators/SmartGlassesFontSize.java
================================================
package com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators;
public enum SmartGlassesFontSize {
SMALL,
MEDIUM,
LARGE
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/smartglassescommunicators/SmartGlassesModes.java
================================================
package com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators;
public enum SmartGlassesModes {
SCROLLING_TEXT_VIEW,
SCREEN_OFF
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/smartglassescommunicators/TextLineSG.java
================================================
package com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators;
public class TextLineSG {
private String text;
private int fontSize;
public TextLineSG(String text, int fontSize){
this.text = text;
this.fontSize = fontSize;
}
public String getText() {
return text;
}
public void setText(String text) {
this.text = text;
}
public int getFontSizeCode() {
return fontSize;
}
public void setFontSize(int fontSize) {
this.fontSize = fontSize;
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/smartglassescommunicators/UltraliteLayoutHelper.java
================================================
package com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators;
import com.vuzix.ultralite.Layout;
public class UltraliteLayoutHelper {
public static String getJsonTag(Layout layout) {
return layout.getJsonTag();
}
public static String getCustomJsonTag(String layout) {
switch (layout) {
case "TEXT_TOP_LEFT_ALIGN":
return "ttla";
default:
return "";
}
}
}
================================================
FILE: SGM_android/SmartGlassesManager/src/main/java/com/teamopensmartglasses/smartglassesmanager/smartglassescommunicators/UltraliteSGC.java
================================================
package com.teamopensmartglasses.smartglassesmanager.smartglassescommunicators;
import android.content.Context;
import android.content.res.Resources;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Typeface;
import android.graphics.drawable.Drawable;
import android.os.Handler;
import android.util.Log;
import androidx.lifecycle.LifecycleOwner;
import androidx.lifecycle.LiveData;
import com.teamopensmartglasses.smartglassesmanager.R;
import com.squareup.picasso.Picasso;
import com.squareup.picasso.Target;
import com.vuzix.ultralite.Anchor;
import com.vuzix.ultralite.EventListener;
import com.vuzix.ultralite.Layout;
import com.vuzix.ultralite.TextAlignment;
import com.vuzix.ultralite.TextWrapMode;
import com.vuzix.ultralite.UltraliteColor;
import com.vuzix.ultralite.UltraliteSDK;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
//communicate with ActiveLook smart glasses
public class UltraliteSGC extends SmartGlassesCommunicator {
private static final String TAG = "WearableAi_UltraliteSGC";
UltraliteSDK ultraliteSdk;
UltraliteSDK.Canvas ultraliteCanvas;
UltraliteListener ultraliteListener;
Layout currentUltraliteLayout;
boolean screenToggleOff = false; //should we keep the screen off?
LifecycleOwner lifecycleOwner;
Context context;
public static final int cardLingerTime = 15;
private ArrayList rowTextsLiveNow;
//ultralite pixel buffer on left side of screen
int ultraliteLeftSidePixelBuffer = 40;
// Constants for maximum lines and characters per line //depends on size of pixel buffer! //for MEDIUM text!
private int maxLines = 12; // Adjusted from 11.5 for practical use
private int maxCharsPerLine = 38; // Assuming max 27 characters fit per line on your display
//handler to turn off screen
Handler goHomeHandler;
Runnable goHomeRunnable;
//handler to turn off screen/toggle
Handler screenOffHandler;
Runnable screenOffRunnable;
//handler to disconnect
Handler killHandler;
boolean hasUltraliteControl;
boolean screenIsClear;
public class UltraliteListener implements EventListener{
@Override
public void onTap(int tapCount) {
Log.d(TAG, "Ultralite go tap n times: " + tapCount);
tapEvent(tapCount);
}
@Override
public void onDisplayTimeout() {
Log.d(TAG, "Ultralite display timeout.");
}
@Override
public void onPowerButtonPress(boolean turningOn) {
//since we implement our own state for the power turn on/off, we ignore what the ultralite thinks ('turningOn') and use our own state
Log.d(TAG, "Ultralites power button pressed: " + turningOn);
//flip value of screen toggle
screenToggleOff = !turningOn;
if (!screenToggleOff) {
Log.d(TAG, "screen toggle off NOT on, showing turn ON message");
displayReferenceCardSimple("SGM Connected.", "Screen back on...", 4);
} else {
Log.d(TAG, "screen toggle off IS on, showing turn OFF message");
// ultraliteCanvas.clear();
// displayReferenceCardSimple("Toggling off....", "Toggling off...", -1);
// screenOffHandler.removeCallbacksAndMessages(this);
// screenOffHandler.removeCallbacksAndMessages(screenOffRunnable);
// screenOffRunnable = new Runnable() {
// @Override
// public void run() {
// ultraliteSdk.screenOff();
// }
// };
// screenOffHandler.postDelayed(screenOffRunnable, 2200);
}
}
}
public UltraliteSGC(Context context, LifecycleOwner lifecycleOwner) {
super();
this.lifecycleOwner = lifecycleOwner;
this.context = context;
mConnectState = 0;
hasUltraliteControl = false;
screenIsClear = true;
goHomeHandler = new Handler();
screenOffHandler = new Handler();
killHandler = new Handler();
rowTextsLiveNow = new ArrayList();
ultraliteSdk = UltraliteSDK.get(context);
ultraliteListener = new UltraliteListener();
ultraliteSdk.addEventListener(ultraliteListener);
LiveData ultraliteConnectedLive = ultraliteSdk.getConnected();
ultraliteConnectedLive.observe(lifecycleOwner, isConnected -> {
onUltraliteConnectedChange(isConnected);
});
LiveData ultraliteControlled = ultraliteSdk.getControlledByMe();
ultraliteControlled.observe(lifecycleOwner, isControlled -> {
onUltraliteControlChanged(isControlled);
});
// if (ultraliteSdk.isAvailable()){
// Log.d(TAG, "Ultralite SDK is available.");
// } else {
// Log.d(TAG, "Ultralite SDK is NOT available.");
// }
}
private void onUltraliteConnectedChange(boolean isConnected) {
Log.d(TAG, "Ultralite CONNECT changed to: " + isConnected);
if (isConnected) {
Log.d(TAG, "Ultralite requesting control...");
boolean isControlled = ultraliteSdk.requestControl();
if (isControlled){
// setupUltraliteCanvas();
changeUltraliteLayout(Layout.CANVAS);
} else {
return;
}
Log.d(TAG, "Ultralite RESULT control request: " + isControlled);
connectionEvent(2);
} else {
Log.d(TAG, "Ultralite not connected.");
connectionEvent(0);
}
}
private void onUltraliteControlChanged(boolean isControlledByMe) {
Log.d(TAG, "Ultralite CONTROL changed to: " + isControlledByMe);
if(isControlledByMe) {
hasUltraliteControl = true;
setupUltraliteCanvas();
connectionEvent(2);
displayReferenceCardSimple("Connected to SGM", "by TeamOpenSmartGlasses", 5);
} else {
hasUltraliteControl = false;
}
// mUltraliteControlledByMe = isControlledByMe;
}
@Override
protected void setFontSizes(){
}
@Override
public void connectToSmartGlasses(){
Log.d(TAG, "connectToSmartGlasses running...");
// int mCount = 10;
// while ((mConnectState != 2) && (!hasUltraliteControl) && (mCount > 0)){
// mCount--;
// try {
// Log.d(TAG, "Don't have Ultralite yet, let's wait for it...");
// Thread.sleep(200);
// } catch (InterruptedException e) {
// e.printStackTrace();
// }
// }
// Log.d(TAG, "Connected to Ultralites.");
// Log.d(TAG, "mCOnnectestate: " + mConnectState);
// Log.d(TAG, "mCOunt: " + mCount);
// displayReferenceCardSimple("Connected to SGM.", "Authors: TeamOpenSmartGlasses");
// connectionEvent(mConnectState);
Log.d(TAG, "connectToSmartGlasses finished");
}
public void displayTextLine(String text){
displayReferenceCardSimple("", text);
}
private static final int MAX_LINES = 7;
public void displayTextWall(String text) {
String cleanedText = cleanText(text);
if (screenToggleOff) {
return;
}
goHomeHandler.removeCallbacksAndMessages(null);
goHomeHandler.removeCallbacksAndMessages(goHomeRunnable);
Log.d(TAG, "Ultralite is doing text wall");
// Cut text wall down to the largest number of lines possible to display
String[] lines = cleanedText.split("\n");
StringBuilder truncatedText = new StringBuilder();
for (int i = 0; i < Math.min(lines.length, MAX_LINES); i++) {
truncatedText.append(lines[i]).append("\n");
}
// changeUltraliteLayout(Layout.TEXT_BOTTOM_LEFT_ALIGN);
changeUltraliteLayout(Layout.TEXT_BOTTOM_LEFT_ALIGN);
ultraliteSdk.sendText(truncatedText.toString().trim());
// changeUltraliteLayout(Layout.CANVAS);
// ultraliteCanvas.removeText(0); //remove last text we added
// Anchor ultraliteAnchor = Anchor.TOP_LEFT;
// TextAlignment ultraliteAlignment = TextAlignment.LEFT;
// int textId = ultraliteCanvas.createText(text, ultraliteAlignment, UltraliteColor.WHITE, ultraliteAnchor, ultraliteLeftSidePixelBuffer, 0, 640 - ultraliteLeftSidePixelBuffer, -1, TextWrapMode.WRAP, true);
//// ultraliteCanvas.createText(title, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_LEFT, ultraliteLeftSidePixelBuffer, 120, 640 - ultraliteLeftSidePixelBuffer, -1, TextWrapMode.WRAP, true);
// Log.d(TAG, "VUZIX TEXT ID: " + textId);
ultraliteCanvas.commit();
screenIsClear = false;
}
private String cleanText(String input) {
// Replace Chinese punctuation with English equivalents
String cleaned = input.replace(" ，", ", ")
.replace("，", ", ")
.replace(" 。", ".")
.replace("。", ".")
.replace(" ！", "!")
.replace(" ？", "?")
.replace("？", "?")
.replace("：", ":")
.replace("；", ";")
.replace("（", "(")
.replace("）", ")")
.replace("【", "[")
.replace("】", "]")
.replace("“", "\"")
.replace("”", "\"")
.replace("、", ",") // No quotes around this one
.replace("‘", "'")
.replace("’", "'");
// Fix contractions: handle spaces around apostrophes
cleaned = cleaned.replaceAll("\\s+'\\s*", "'");
// Remove any non-breaking spaces and trim leading/trailing spaces
// cleaned = cleaned.replace("\u00A0", " ").trim();
return cleaned;
}
public static int countNewLines(String str) {
int count = 0;
for (int i = 0; i < str.length(); i++) {
if (str.charAt(i) == '\n') {
count++;
}
}
return count;
}
public void displayDoubleTextWall(String textTop, String textBottom) {
if (screenToggleOff) {
return;
}
textTop = cleanText(textTop);
textBottom = cleanText(textBottom);
// if (textBottom.endsWith("\n")) {
// textBottom = textBottom.substring(0, textBottom.length() - 1);
// }
goHomeHandler.removeCallbacksAndMessages(null);
goHomeHandler.removeCallbacksAndMessages(goHomeRunnable);
// int rowsTop = 5;
int rowsTop = 5 - countNewLines(textTop);
StringBuilder combinedText = new StringBuilder();
combinedText.append(textTop);
for (int i = 0; i < rowsTop; i++) {
combinedText.append("\n");
}
StringBuilder bottomBuilder = new StringBuilder(textBottom);
combinedText.append(bottomBuilder);
// Display the combined text using TEXT_BOTTOM_LEFT_ALIGN layout
changeUltraliteLayout(Layout.TEXT_BOTTOM_LEFT_ALIGN);
// ultraliteSdk.sendText(combinedText.toString().trim());
ultraliteSdk.sendText(combinedText.toString());
ultraliteCanvas.commit();
screenIsClear = false;
}
public void displayCenteredText(String text){
}
public void displayCustomContent(String json) {
displayReferenceCardSimple("CustomDisplayNotImplemented", json);
}
public void showNaturalLanguageCommandScreen(String prompt, String naturalLanguageInput){
// int boxDelta = 3;
//
// if (connectedGlasses != null) {
// connectedGlasses.clear();
// showPromptCircle();
//
// //show the prompt
// lastLocNaturalLanguageArgsTextView = displayText(new TextLineSG(prompt, SMALL_FONT), new Point(0, 11), true);
// lastLocNaturalLanguageArgsTextView = new Point(lastLocNaturalLanguageArgsTextView.x, lastLocNaturalLanguageArgsTextView.y + boxDelta); //margin down a tad
//
// //show the final "finish command" prompt
// int finishY = 90;
// displayLine(new Point(0, finishY), new Point(100, finishY));
// displayText(new TextLineSG(finishNaturalLanguageString, SMALL_FONT), new Point(0, finishY + 2), true);
//
// //show the natural language args in a scroll box
//// ArrayList nli = new ArrayList<>();
//// nli.add(new TextLineSG(naturalLanguageInput, SMALL_FONT));
//// lastLocNaturalLanguageArgsTextView = scrollTextShow(nli, startScrollBoxY.y + boxDelta, finishY - boxDelta);
// }
}
public void updateNaturalLanguageCommandScreen(String naturalLanguageArgs){
// Log.d(TAG, "Displaynig: " + naturalLanguageArgs);
// displayText(new TextLineSG(naturalLanguageArgs, SMALL_FONT), new Point(0, lastLocNaturalLanguageArgsTextView.y));
}
public void blankScreen(){
// if (connectedGlasses != null){
// connectedGlasses.clear();
// }
}
@Override
public void destroy(){
if (ultraliteSdk != null){
// displayReferenceCardSimple("Disconnecting...", "Disconnecting Smart Glasses from SGM");
//
// //disconnect after slight delay, so our above text gets a chance to show up
// killHandler.postDelayed(new Runnable() {
// @Override
// public void run() {
// ultraliteSdk.releaseControl();
// }
// }, 800);
ultraliteSdk.removeEventListener(ultraliteListener);
ultraliteSdk.releaseControl();
}
}
public void showHomeScreen(){
changeUltraliteLayout(Layout.CANVAS);
ultraliteCanvas.clear();
screenIsClear = true;
}
public void setupUltraliteCanvas(){
Log.d(TAG, "Setting up ultralite canvas");
if (ultraliteSdk != null) {
ultraliteCanvas = ultraliteSdk.getCanvas();
}
}
public void changeUltraliteLayout(Layout chosenLayout) {
//don't update layout if it's already setup
if (currentUltraliteLayout != null && currentUltraliteLayout == chosenLayout){
return;
}
currentUltraliteLayout = chosenLayout;
ultraliteSdk.setLayout(chosenLayout, 0, true);
if (chosenLayout.equals(Layout.CANVAS)){
if (ultraliteCanvas == null){
setupUltraliteCanvas();
}
}
}
public void startScrollingTextViewMode(String title){
super.startScrollingTextViewMode(title);
if (ultraliteSdk == null) {
return;
}
//clear the screen
ultraliteCanvas.clear();
drawTextOnUltralite(title);
}
public String addNewlineEveryNWords(String input, int n) {
String[] words = input.split("\\s+");
StringBuilder result = new StringBuilder();
for (int i = 0; i < words.length; i++) {
result.append(words[i]);
if ((i + 1) % n == 0 && i != words.length - 1) {
result.append("\n");
} else if (i != words.length - 1) {
result.append(" ");
}
}
return result.toString();
}
public void drawTextOnUltralite(String text){
//edit the text to add new lines to it because ultralite wrapping doesn't work
String wrappedText = addNewlineEveryNWords(text, 6);
//display the title at the top of the screen
UltraliteColor ultraliteColor = UltraliteColor.WHITE;
Anchor ultraliteAnchor = Anchor.TOP_LEFT;
TextAlignment ultraliteAlignment = TextAlignment.LEFT;
changeUltraliteLayout(Layout.CANVAS);
ultraliteCanvas.clear();
ultraliteCanvas.clearBackground(UltraliteColor.DIM);
// ultraliteCanvas.createText(text, ultraliteAlignment, ultraliteColor, ultraliteAnchor, true);
// ultraliteCanvas.createText(text, ultraliteAlignment, ultraliteColor, Anchor.BOTTOM_LEFT, 0, 0, -1, 80, TextWrapMode.WRAP, true);
ultraliteCanvas.createText(wrappedText, ultraliteAlignment, ultraliteColor, ultraliteAnchor, true); //, 0, 0, -1, -1, TextWrapMode.WRAP, true);
ultraliteCanvas.commit();
screenIsClear = false;
}
public Bitmap getBitmapFromDrawable(Resources res) {
return BitmapFactory.decodeResource(res, R.drawable.vuzix_shield);
}
public void displayReferenceCardSimple(String title, String body){
displayReferenceCardSimple(title, body, cardLingerTime);
}
// public void displayReferenceCardSimple(String title, String body, int lingerTime){
// if (!isConnected()) {
// Log.d(TAG, "Not showing reference card because not connected to Ultralites...");
// return;
// }
//
//// String [] bulletPoints = {"first one", "second one", "dogs and cats"};
//// displayBulletList("Cool Bullets:", bulletPoints, 15);
//
// Log.d(TAG, "Sending text to Ultralite SDK: \n" + body);
//// ultraliteSdk.sendText("hello world"); //this is BROKEN in Vuzix ultralite 0.4.2 SDK - crashes Vuzix OEM Platform android app
//
// //edit the text to add new lines to it because ultralite wrapping doesn't work
//// String titleWrapped = addNewlineEveryNWords(title, 6);
//// String bodyWrapped = addNewlineEveryNWords(body, 6);
//
// //display the title at the top of the screen
// UltraliteColor ultraliteColor = UltraliteColor.WHITE;
// Anchor ultraliteAnchor = Anchor.TOP_LEFT;
// TextAlignment ultraliteAlignment = TextAlignment.LEFT;
// changeUltraliteLayout(Layout.CANVAS);
// ultraliteCanvas.clear();
// ultraliteCanvas.createText(title, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_LEFT, ultraliteLeftSidePixelBuffer, 120, 640 - ultraliteLeftSidePixelBuffer, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.createText(body, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.MIDDLE_LEFT, ultraliteLeftSidePixelBuffer, 0, 640 - ultraliteLeftSidePixelBuffer, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.commit();
// screenIsClear = false;
//
// homeScreenInNSeconds(lingerTime);
// }
public void setFontSize(SmartGlassesFontSize fontSize){
int textSize;
switch (fontSize){
case SMALL:
textSize = 24;
maxLines = 14;
maxCharsPerLine = 42;
break;
case MEDIUM:
textSize = 29;
maxLines = 12; // Adjusted from 11.5 for practical use
maxCharsPerLine = 38; // Assuming max 27 characters fit per line on your display
break;
case LARGE:
textSize = 40;
maxLines = 7;
maxCharsPerLine = 28;
break;
default:
throw new IllegalArgumentException("Unknown font size: " + fontSize);
}
ultraliteSdk.setFont(null, 0, textSize);
}
public void displayReferenceCardSimple(String titleStr, String bodyStr, int lingerTime){
if (screenToggleOff){
return;
}
String title = maybeReverseRTLString(titleStr);
String body = maybeReverseRTLString(bodyStr);
if (!isConnected()) {
Log.d(TAG, "Not showing reference card because not connected to Ultralites...");
return;
}
changeUltraliteLayout(Layout.CANVAS);
ultraliteCanvas.clear();
// String [] bulletPoints = {"first one", "second one", "dogs and cats"};
// displayBulletList("Cool Bullets:", bulletPoints, 15);
Log.d(TAG, "Sending text to Ultralite SDK: \n" + body);
// ultraliteSdk.sendText("hello world"); //this is BROKEN in Vuzix ultralite 0.4.2 SDK - crashes Vuzix OEM Platform android app
//edit the text to add new lines to it because ultralite wrapping doesn't work
// String titleWrapped = addNewlineEveryNWords(title, 6);
// String bodyWrapped = addNewlineEveryNWords(body, 6);
//display title top of scren adn text middle of screen
// UltraliteColor ultraliteColor = UltraliteColor.WHITE;
// Anchor ultraliteAnchor = Anchor.TOP_LEFT;
// TextAlignment ultraliteAlignment = TextAlignment.LEFT;
// ultraliteCanvas.createText(title, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_LEFT, ultraliteLeftSidePixelBuffer, 120, 640 - ultraliteLeftSidePixelBuffer, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.createText(body, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.MIDDLE_LEFT, ultraliteLeftSidePixelBuffer, 0, 640 - ultraliteLeftSidePixelBuffer, -1, TextWrapMode.WRAP, true);
//concat body and title, put text on top right of screen (to not block main view)
UltraliteColor ultraliteColor = UltraliteColor.WHITE;
Anchor ultraliteAnchor = Anchor.TOP_CENTER;
TextAlignment ultraliteAlignment = TextAlignment.LEFT;
//ultraliteCanvas.createText(body, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_RIGHT, 0, 0, (640 / 2) - ultraliteLeftSidePixelBuffer, -1, TextWrapMode.WRAP, true);
if (!title.isEmpty() && !title.equals("")){
ultraliteCanvas.createText(title + ": " + body, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_RIGHT, 0, 0, 640 / 2, -1, TextWrapMode.WRAP, true);
} else {
ultraliteCanvas.createText(body, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_RIGHT, 0, 0, 640 / 2, -1, TextWrapMode.WRAP, true);
}
//NOTE:
// int createText(@NonNull
// String text,
// @NonNull
// TextAlignment alignment,
// @NonNull
// UltraliteColor color,
// @NonNull
// Anchor anchor,
// int offsetX,
// int offsetY,
// int width,
// int height,
// @Nullable
// TextWrapMode wrap,
// boolean visible)
ultraliteCanvas.commit();
screenIsClear = false;
homeScreenInNSeconds(lingerTime);
}
public void displayBulletList(String title, String [] bullets){
displayBulletList(title, bullets, 14);
}
public void displayRowsCard(String[] rowStrings){
displayRowsCard(rowStrings, cardLingerTime);
}
public void displayRowsCard(String[] rowStringList, int lingerTime){
if (screenToggleOff){
return;
}
String[] rowStrings = maybeReverseRTLStringList(rowStringList);
if (!isConnected()) {
Log.d(TAG, "Not showing rows card because not connected to Ultralites...");
return;
}
// changeUltraliteLayout(Layout.CANVAS);
// ultraliteCanvas.clear();
//make lines to draw on screen to delineate rows
int line_thickness = 3;
for (int y = 120; y < 480; y += 120) {
ultraliteCanvas.clearBackgroundRect(0, y, 640, line_thickness, UltraliteColor.DIM);
}
//clear old text
for (int i = 0; i < rowTextsLiveNow.size(); i++){
ultraliteCanvas.removeText(i);
}
//old way to clear old text - vuzix ultralite sdk bug that clear background doesn't clear text?
// for (int y = 0; y < 480; y += 120) {
// //clear previous text
// ultraliteCanvas.clearBackgroundRect(0, y + line_thickness, 640, 120 - line_thickness, UltraliteColor.DIM);
// ultraliteCanvas.clearBackgroundRect(0, y + line_thickness, 640, 120 - line_thickness, UltraliteColor.BLACK);
// }
// ultraliteCanvas.commit();
//display the title at the top of the screen
UltraliteColor ultraliteColor = UltraliteColor.WHITE;
Anchor ultraliteAnchor = Anchor.TOP_LEFT;
TextAlignment ultraliteAlignment = TextAlignment.LEFT;
//if no input, just show the lines
if (rowStrings.length == 0){
ultraliteCanvas.commit();
screenIsClear = false;
// homeScreenInNSeconds(lingerTime);
return;
}
//go throw rows, draw the text, don't do more than 4
int y_start_height = 55;
// Reverse rowStrings array
Collections.reverse(Arrays.asList(rowStrings));
int numRows = 4;
int actualRows = Math.min(rowStrings.length, numRows);
for (int i = 0; i < actualRows; i++) {
// Calculate the offset to start from the bottom for 1, 2, or 3 values
int yOffset = (numRows - actualRows) * 112;
int textId = ultraliteCanvas.createText(rowStrings[i], TextAlignment.CENTER, UltraliteColor.WHITE, Anchor.TOP_LEFT, ultraliteLeftSidePixelBuffer, y_start_height + yOffset + (i * 112), 640 - ultraliteLeftSidePixelBuffer, -1, TextWrapMode.WRAP, true);
rowTextsLiveNow.add(textId);
}
ultraliteCanvas.commit();
screenIsClear = false;
// homeScreenInNSeconds(cardLingerTime);
}
public void displayBulletList(String title, String [] bulletList, int lingerTime){
if (screenToggleOff){
return;
}
String[] bullets = maybeReverseRTLStringList(bulletList);
if (!isConnected()) {
Log.d(TAG, "Not showing bullet point list because not connected to Ultralites...");
return;
}
Log.d(TAG, "Sending bullets to Ultralite SDK: " + title);
//display the title at the top of the screen
UltraliteColor ultraliteColor = UltraliteColor.WHITE;
Anchor ultraliteAnchor = Anchor.TOP_LEFT;
TextAlignment ultraliteAlignment = TextAlignment.LEFT;
changeUltraliteLayout(Layout.CANVAS);
ultraliteCanvas.clear();
ultraliteCanvas.createText(title, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_LEFT, 0, 0, 640, -1, TextWrapMode.WRAP, true);
int displaceY = 25;
int displaceX = 25;
for (String bullet : bullets){
ultraliteCanvas.createText("⬤ " + bullet, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_LEFT, displaceX, displaceY, 640 - displaceX, -1, TextWrapMode.WRAP, true);
displaceY += 125;
}
ultraliteCanvas.commit();
screenIsClear = false;
// if (lingerTime > 0){
// homeScreenInNSeconds(lingerTime);
// }
}
public void homeScreenInNSeconds(int n){
if (n == -1){
return;
}
//disconnect after slight delay, so our above text gets a chance to show up
goHomeHandler.removeCallbacksAndMessages(null);
goHomeHandler.removeCallbacksAndMessages(goHomeRunnable);
goHomeRunnable = new Runnable() {
@Override
public void run() {
showHomeScreen();
}};
goHomeHandler.postDelayed(goHomeRunnable, n * 1000);
}
public void displayBitmap(Bitmap bmp) {
Bitmap resizedBmp = Bitmap.createScaledBitmap(bmp, 620, 460, true); // 640 x 480
changeUltraliteLayout(Layout.CANVAS);
screenIsClear = false;
Log.d(TAG, "Sending bitmap to Ultralite");
ultraliteCanvas.drawBackground(resizedBmp, 50, 80);
ultraliteCanvas.commit();
}
//don't show images on activelook (screen is too low res)
public void displayReferenceCardImage(String title, String body, String imgUrl){
if (screenToggleOff){
return;
}
changeUltraliteLayout(Layout.CANVAS);
ultraliteCanvas.clear();
//make image
//below works, but only for very, very low res/size images
Anchor ultraliteImageAnchor = Anchor.CENTER;
Picasso.get()
.load(imgUrl)
.into(new Target() {
@Override
public void onBitmapLoaded(Bitmap bitmap, Picasso.LoadedFrom from) {
// Use the bitmap
// LVGLImage ultraliteImage = LVGLImage.fromBitmap(getBitmapFromDrawable(context.getResources()), CF_INDEXED_2_BIT);
// LVGLImage ultraliteImage = LVGLImage.fromBitmap(bitmap, CF_INDEXED_2_BIT);
changeUltraliteLayout(Layout.CANVAS);
//send text first, cuz this is fast
ultraliteCanvas.createText(title, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.TOP_LEFT, 0, 0, 640, -1, TextWrapMode.WRAP, true);
ultraliteCanvas.createText(body, TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.BOTTOM_LEFT, 0, 0, 640, -1, TextWrapMode.WRAP, true);
ultraliteCanvas.commit();
screenIsClear = false;
Log.d(TAG, "Sending image to Ultralite");
// ultraliteCanvas.createImage(ultraliteImage, ultraliteImageAnchor, 0, 0, true);
ultraliteCanvas.drawBackground(bitmap, 0, 0);
//sending text again to ultralite in case image overwrote it
// ultraliteCanvas.createText(title + "2", TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.BOTTOM_LEFT, 0, 0, 640, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.createText(body + "2", TextAlignment.AUTO, UltraliteColor.WHITE, Anchor.MIDDLE_LEFT, 0, 0, 640, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.commit();
// //display the title at the top of the screen
// UltraliteColor ultraliteColor = UltraliteColor.WHITE;
// TextAlignment ultraliteAlignment = TextAlignment.LEFT;
// // ultraliteCanvas.clearBackground(UltraliteColor.DIM);
// ultraliteCanvas.createText(titleWrapped, ultraliteAlignment, ultraliteColor, Anchor.TOP_LEFT, true); //, 0, 0, -1, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.createText(bodyWrapped, ultraliteAlignment, ultraliteColor, Anchor.BOTTOM_LEFT, true); //, 0, 0, -1, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.commit();
// homeScreenInNSeconds(cardLingerTime);
}
@Override
public void onBitmapFailed(Exception e, Drawable errorDrawable) {
// Handle the error
Log.d(TAG, "Bitmap failed");
e.printStackTrace();
}
@Override
public void onPrepareLoad(Drawable placeHolderDrawable) {
// Called before the image is loaded. You can set a placeholder if needed.
}
});
//edit the text to add new lines to it because ultralite wrapping doesn't work
// String titleWrapped = addNewlineEveryNWords(title, 6);
// String bodyWrapped = addNewlineEveryNWords(body, 6);
//
// //display the title at the top of the screen
// UltraliteColor ultraliteColor = UltraliteColor.WHITE;
// TextAlignment ultraliteAlignment = TextAlignment.LEFT;
// //ultraliteCanvas.clearBackground(UltraliteColor.DIM);
// ultraliteCanvas.createText(titleWrapped, ultraliteAlignment, ultraliteColor, Anchor.TOP_LEFT, true); //, 0, 0, -1, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.createText(bodyWrapped, ultraliteAlignment, ultraliteColor, Anchor.BOTTOM_LEFT, true); //, 0, 0, -1, -1, TextWrapMode.WRAP, true);
// ultraliteCanvas.commit();
// screenIsClear = false;
}
//handles text wrapping, returns final position of last line printed
// private Point displayText(TextLineSG textLine, Point percentLoc, boolean centered){
// if (!isConnected()){
// return null;
// }
//
// //get info about the wrapping
// Pair wrapInfo = computeStringWrapInfo(textLine);
// int numWraps = (int)wrapInfo.first;
// int wrapLenNumChars = (int)wrapInfo.second;
//
// //loop through the text, writing out individual lines to the glasses
// ArrayList chunkedText = new ArrayList<>();
// Point textPoint = percentLoc;
// int textMarginY = computeMarginPercent(textLine.getFontSizeCode()); //(fontToSize.get(textLine.getFontSize()) * 1.3)
// for (int i = 0; i <= numWraps; i++){
// int startIdx = wrapLenNumChars * i;
// int endIdx = Math.min(startIdx + wrapLenNumChars, textLine.getText().length());
// String subText = textLine.getText().substring(startIdx, endIdx).trim();
// chunkedText.add(subText);
// TextLineSG thisTextLine = new TextLineSG(subText, textLine.getFontSizeCode());
// if (!centered) {
// sendTextToGlasses(thisTextLine, textPoint);
// } else {
// int xPercentLoc = computeStringCenterInfo(thisTextLine);
// sendTextToGlasses(thisTextLine, new Point(xPercentLoc, textPoint.y));
// }
// textPoint = new Point(textPoint.x, textPoint.y + pixelToPercent(displayHeightPixels, fontToSize.get(textLine.getFontSizeCode())) + textMarginY); //lower our text for the next loop
// }
//
// return textPoint;
// }
public void stopScrollingTextViewMode() {
// if (connectedGlasses == null) {
// return;
// }
//
// //clear the screen
// connectedGlasses.clear();
}
public void scrollingTextViewIntermediateText(String text){
}
public void scrollingTextViewFinalText(String text){
if (!isConnected()){
return;
}
// //save to our saved list of final scrolling text strings
// finalScrollingTextStrings.add(text);
//
// //get the max number of wraps allows
// float allowedTextRows = computeAllowedTextRows(fontToSize.get(scrollingTextTitleFontSize), fontToSize.get(scrollingTextTextFontSize), percentToPixel(displayHeightPixels, computeMarginPercent(scrollingTextTextFontSize)));
//
// //figure out the maximum we can display
// int totalRows = 0;
// ArrayList finalTextToDisplay = new ArrayList<>();
// boolean hitBottom = false;
// for (int i = finalScrollingTextStrings.toArray().length - 1; i >= 0; i--){
// String finalText = finalScrollingTextStrings.get(i);
// //convert to a TextLine type with small font
// TextLineSG tlString = new TextLineSG(finalText, SMALL_FONT);
// //get info about the wrapping of this string
// Pair wrapInfo = computeStringWrapInfo(tlString);
// int numWraps = (int)wrapInfo.first;
// int wrapLenNumChars = (int)wrapInfo.second;
// totalRows += numWraps + 1;
//
// if (totalRows > allowedTextRows){
// finalScrollingTextStrings = finalTextToDisplay;
// lastLocScrollingTextView = belowTitleLocScrollingTextView;
// //clear the glasses as we hit our limit and need to redraw
// connectedGlasses.color((byte)0x00);
// connectedGlasses.rectf(percentScreenToPixelsLocation(belowTitleLocScrollingTextView.x, belowTitleLocScrollingTextView.y), percentScreenToPixelsLocation(100, 100));
// //stop looping, as we've ran out of room
// hitBottom = true;
// } else {
// finalTextToDisplay.add(0, finalText);
// }
// }
//
// //display all of the text that we can
// if (hitBottom) { //if we ran out of room, we need to redraw all the text
// for (String finalString : finalTextToDisplay) {
// TextLineSG tlString = new TextLineSG(finalString, scrollingTextTextFontSize);
// //write this text at the last location + margin
// Log.d(TAG, "Writing string: " + tlString.getText() + finalTextToDisplay.size());
// lastLocScrollingTextView = displayText(tlString, new Point(0, lastLocScrollingTextView.y));
// }
// } else { //if we didn't hit the bottom, and there's room, we can just display the next line
// TextLineSG tlString = new TextLineSG(text, scrollingTextTextFontSize);
// lastLocScrollingTextView = displayText(tlString, new Point(0, lastLocScrollingTextView.y));
// }
}
public static String maybeReverseRTLString(String text) {
StringBuilder result = new StringBuilder();
StringBuilder rtlBuffer = new StringBuilder();
for (char c : text.toCharArray()) {
if (isRTLCharacter(c)) {
rtlBuffer.append(c); // Append RTL characters to a buffer
} else {
if (rtlBuffer.length() > 0) {
result.append(rtlBuffer.reverse()); // Reverse and append RTL text when a non-RTL character is found
rtlBuffer.setLength(0); // Clear the buffer
}
result.append(c); // Append non-RTL characters directly to the result
}
}
if (rtlBuffer.length() > 0) {
result.append(rtlBuffer.reverse()); // Append any remaining RTL text in reverse
}
return result.toString();
}
private static boolean isRTLCharacter(char c) {
Character.UnicodeBlock block = Character.UnicodeBlock.of(c);
return block == Character.UnicodeBlock.ARABIC ||
block == Character.UnicodeBlock.HEBREW ||
block == Character.UnicodeBlock.SYRIAC ||
block == Character.UnicodeBlock.ARABIC_SUPPLEMENT ||
block == Character.UnicodeBlock.THAANA ||
block == Character.UnicodeBlock.NKO ||
block == Character.UnicodeBlock.SAMARITAN ||
block == Character.UnicodeBlock.MANDAIC ||
block == Character.UnicodeBlock.ARABIC_EXTENDED_A;
// Add other RTL blocks as needed
}
public String[] maybeReverseRTLStringList(String[] in){
String[] out = new String[in.length];
for(int i = 0; i < in.length; i++)
out[i] = maybeReverseRTLString(in[i]);
return out;
}
public void displayPromptView(String prompt, String [] options){
if (!isConnected()){
return;
}
// ultraliteCanvas.clear();
// connectedGlasses.clear();
// showPromptCircle();
//
// //show the prompt and options, if any
// ArrayList
