Still dead in the water. The

Joe_C_3 · ‎05-30-2015

Hi all, I am attempting to get the Intel RealSense voice synthesiser to work but Unity 5 crashes on my almost every time i run my script. I have an i7-4790k, 8gb ram, gtx-960 and an ssd.

I have all of the correct DLLs installed for a 64bit environment.

This is the script I'm using to test the voice synth, keep in mind that I'm new to c# so I might have an error in my code, though I don't think this is the issue as the debugger isn't showing any.

using UnityEngine;
using System.Collections;
using RSUnityToolkit;
using System.Threading;

public static class Speaker
{
	public static void Speak()
	{
		PXCMSession session = PXCMSession.CreateInstance();
		
		PXCMSpeechSynthesis tts;
		session.CreateImpl<PXCMSpeechSynthesis>(out tts);
		PXCMSpeechSynthesis.ProfileInfo pinfo;
		tts.QueryProfile(0,out pinfo);
		pinfo.language=PXCMSpeechSynthesis.LanguageType.LANGUAGE_US_ENGLISH;
		tts.SetProfile(pinfo);

		tts.BuildSentence(1, "Speak this");

		int nbuffers=tts.QueryBufferNum(1);

		for (int i=0;i<nbuffers;i++) {
		   PXCMAudio audio=tts.QueryBuffer(1, i);
		}

		tts.ReleaseSentence(1);
	}
};

public class Speech : MonoBehaviour {
	void Start () {
		Thread oThread = new Thread(new ThreadStart(Speaker.Speak));
		oThread.Start();
		while (!oThread.IsAlive);
		oThread.Abort();
	}
}

Joe_C_3 · ‎05-30-2015

I was able to work around this by re-installing some drivers for my computer. Unity no longer seems to crash, however this line is no good:

PXCMSession session = PXCMSession.CreateInstance();

as PXCMSession.CreateInstance is returning null and making the rest of the script fail.

Joe_C_3 · ‎05-31-2015

Still dead in the water. The script plays no audio and Unity still crashes.

Threading (provided I was doing it correctly) did not solve the crashing issue as people in other forums have suggested the solution to be (I was able to get the program to crash a few times just moments ago by running a scene and stopping it around 3 times - this does not happen with any of my other projects).

Here's more info about my system:

originally I had an AMD processor and board installed not realizing that this SDK would only work on an intel processor. Stupid mistake but it gave me a good excuse to upgrade so I slapped an i7-4790k and a new motherboard (asrock 287 extreme4). Windows 8.1 worked without re-install, but I decided to wipe everything and start fresh to be safe.
all windows 8.1 updates to-date have been installed
unity 5 is the latest version
I do not own a web cam and did get some warnings when installing the SDK - I am only after the voice synthesis portion of the SDK though so I don't this this is an issue.

If anyone has any extra info they could provide me with that would be amazing - one of my projects is pretty dependent on speech synthesis and this seems to be the best solution out there (if I could get it to work). Here is the updated script:

using UnityEngine;
using System.Collections;
using RSUnityToolkit;
using System.Threading;

public static class Speaker
{
	public static void Speak()
	{
		PXCMSession session = PXCMSession.CreateInstance();
		
		PXCMSpeechSynthesis tts;
		session.CreateImpl<PXCMSpeechSynthesis>(out tts);
		PXCMSpeechSynthesis.ProfileInfo pinfo;
		tts.QueryProfile(0,out pinfo);
		pinfo.language=PXCMSpeechSynthesis.LanguageType.LANGUAGE_US_ENGLISH;
		tts.SetProfile(pinfo);

		tts.BuildSentence(1, "Speak this");

		int nbuffers=tts.QueryBufferNum(1);

		for (int i=0;i<nbuffers;i++) {
		   PXCMAudio audio=tts.QueryBuffer(1, i);
		}

		tts.ReleaseSentence(1);
	}
};

public class Speech : MonoBehaviour {
	void Start () {
		Thread oThread = new Thread(new ThreadStart(Speaker.Speak));
		oThread.Start();
		while (!oThread.IsAlive);
		oThread.Abort();
	}
}

Joe_C_3 · ‎05-31-2015

More info -

realized that the audio isn't just going to magically play in Unity. Attached an audio source to my game object with the intention of attaching an audio clip created by the SDK to it, however when I attempt to create a Unity AudioClip (PXCMSpeechSynthesis.CreateAudioClip as per the documentation I get this error in Unity:

The name `CreateAudioClip' does not exist in the current context

So rather than using AudioClip ac = CreateAudioClip(1); I tried using AudioClip ac = PXCMSpeechSynthesis.CreateAudioClip(1); but this also fails with the error: `PXCMSpeechSynthesis' does not contain a definition for `CreateAudioClip'

MartyG · ‎06-01-2015

I had a dead camera in Unity 5 when doing a full Build and Run or running a published standalone executable. I found that the camera worked once I went to File > Build Settings and, in the Architecture menu of the Build Settings pop-up window, selected the 'X86_64' option to designate that Unity should compile the program in 64 bit to match the 64-bit RealSense drivers used in Unity.

Doing this might not make a difference to the audio problem you are experiencing. Then again, it only takes 10 seconds to make the change and you'll probably have to do the setting anyway at some point, so why not give it a go now. :)

Joe_C_3 · ‎06-02-2015

Thanks, Marty - I made the switch which will help with compiled builds no doubt. I just decided to take another look at this after stepping away for a bit. I think the issue really does surround CreateAudioClip not existing in the SDK.

Take a look at this screenshot - the only reference to CreateAudioClip that is found in a case insensetive search through all of C:\Program Files (x86)\Intel\RSSDK is in the documentation.

I suppose I just need to figure out how to create a unity audio clip out of the sentence buffer myself, though I wish the documentation didn't mention it if it really doesn't exist.

MartyG · ‎06-02-2015

The online documentation for RealSense has the following info about createaudioclip, though it's not much and seems to be the same one that is listed on your locally stored docs as 'createaudioclip_pxcmspeechsynthesis.html'.

https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/createaudioclip_pxcmspeechsynthesis.html

This was the only example I could find of createaudioclip being mentioned despite an extensive internet search.

Joe_C_3 · ‎06-02-2015

Yes sadly that's the same reference as the local docs. I wasn't able to find anything in searches either. Very frustrating. If I can figure out how to create an audio clip from the sentence buffer I'll post the solution.

Joe_C_3 · ‎06-02-2015

not sure if i'm going about this the right way yet, but figured I'd throw this here for the dev team (if they're paying attention to this).

more bad docs:

PXCMAudio.Access.ACCESS_READ as shown here:

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/acquireaccess_pxcaudio.html

should actually be PXCMAudio.Access.READ_ACCESS as shown here:

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/access_pxcaudio.html

Joe_C_3 · ‎06-02-2015

ok i finally found this file

C:\Program Files (x86)\Intel\RSSDK\framework\common\pxcclr.cs\src\pxcmspeechsynthesis-audioclip.cs

which contains a complete example of extracting the audio clip from the the buffer and attaching it to an audio source. glad i was on the right track, not so glad it took me so long to find this, but I have to absorb some of the blame as I should have searched "AudioClip" within the imported project files right away before digging through documentation- I had tunnel vision for a while. At least I gained a better understanding of the lib. I'll report back with results later today. I have a feeling I'll need to modify the example script to work with a thread to prevent crashing.

Lance_R_ · ‎06-05-2015

For one thing, Joe, I'd like to point out that you're using the wrong documentation pages and that the "v1.1" docs are not up to date apparently. The most recent documentation for the "gold2014" SDK can be found here:

(PXCMAudio >> Member Functions >> Acquire Access)

https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/acquireaccess_pxcaudio.html

(PXCMAudio >> Enumerators >> Access)

https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/access_pxcaudio.html

I don't know if adding the C# extensions file will help... as described here:

https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/devguide_unity__application.html

But it contains a few additional audio format methods.

"Copy the following file(s) from... $(RSSDK_DIR)/framework/common/pxcclr.cs/src: ...

pxcmdefs.extensions.cs [and add it to your Plugins.Managed folder in Unity 5]

This optional file provides additional type extension functions."

Joe_C_3 · ‎06-09-2015

Lance R. wrote:

For one thing, Joe, I'd like to point out that you're using the wrong documentation pages and that the "v1.1" docs are not up to date apparently. The most recent documentation for the "gold2014" SDK can be found here:

Wow I feel like an idiot - I didn't realize I was using outdated docs. I had to put this project to the side for the last few days due to health issues but will be resuming within the week. Thanks so much for the link - will report back once I have implemented it.

Colleen_C_Intel · ‎06-09-2015

I'm confused now too ecause I believe that the v1.1 are now the latest set of docs (the actual 1.1 were overwritten with the R3 release set) making the 2014gold an older set (R2).
https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/index.html?https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?

Can you clarify what was correct in 2014 gold that wasn't in v1.1?

Joe_C_3 · ‎06-10-2015

Colleen Culbertson (Intel) wrote:

I'm confused now too ecause I believe that the v1.1 are now the latest set of docs (the actual 1.1 were overwritten with the R3 release set) making the 2014gold an older set (R2).
https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/index.html?https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?

Can you clarify what was correct in 2014 gold that wasn't in v1.1?

Being that you are posted the articles that I followed to the documentation in the first place and you work at Intel, I'm guessing that you are correct. I am hoping to get this solved on my end today (this weekend at the absolute latest - all depends on how much time becomes available). I will absolutely post results once I have them. I hate finding old threads that never get solved.

Joe_C_3 · ‎06-12-2015

First off as promised, a full working example. To use, create a Gameobject, attach an AudioSource along with this script:

using System;
using System.Runtime.InteropServices;
using UnityEngine;
using System.Collections;

public class SpeechTest : MonoBehaviour {
	private static PXCMSpeechSynthesis tts;
	private static AudioClip clip;
	private static PXCMAudio audio;

	void Start () {
		int nbuffers;
		int nsamples;
		AudioSource aud = GetComponent<AudioSource>();
		String name = "test";
		Int32 sid = 1;
		PXCMSession session = PXCMSession.CreateInstance();

		session.CreateImpl<PXCMSpeechSynthesis>(out tts);
		PXCMSpeechSynthesis.ProfileInfo pinfo;
		tts.QueryProfile(0,out pinfo);
		pinfo.language=PXCMSpeechSynthesis.LanguageType.LANGUAGE_US_ENGLISH;
		tts.SetProfile(pinfo);
		tts.BuildSentence(1, "This is finally working!");

		nsamples = tts.QuerySampleNum(sid);

		if (nsamples!=0) {
            tts.QueryProfile(out pinfo);
	        clip = AudioClip.Create(
	        	name,
	        	nsamples * pinfo.outputs.nchannels,
	            pinfo.outputs.nchannels,
	            pinfo.outputs.sampleRate,
	            false,
	            false
	        );

	        nbuffers = tts.QueryBufferNum(sid);

	        for (int i = 0, offset=0; i < nbuffers; i++)
	        {
	            PXCMAudio audio = tts.QueryBuffer(sid, i);
	            PXCMAudio.AudioData data;
	            pxcmStatus sts=audio.AcquireAccess(
	            	PXCMAudio.Access.ACCESS_READ,
	            	PXCMAudio.AudioFormat.AUDIO_FORMAT_IEEE_FLOAT,
	            	out data
	            );

	            if (sts < pxcmStatus.PXCM_STATUS_NO_ERROR) break;

	            float[] samples=data.ToFloatArray();

	            clip.SetData(data.ToFloatArray(), offset);

	            offset += samples.Length;

	            audio.ReleaseAccess(data);
	        }

	        aud.clip = clip;
	        aud.Play();
        }
	}
}

Sadly this will only work the first time you play the scene. If you stop the scene and play it again, Unity will crash.

In this forum another user (Joel C) suggests that he was able to get this running by placing the speech synthesis in another thread. This does not work for me, Unity still crashes with the following:

libpxctts_ve_nuance.dll caused an Access Violation (0xc0000005)
in module libpxctts_ve_nuance.dll at 0033:cf07dace.

as an aside that same user mentions using voice_synthesis.cs.VoiceSynthesis.Speak and refers to it as being part of the Unity framework scripts. This is not (or at least is no longer) true - that script can be found under CSharp\FF_SpeechSynthesis.cs\VoiceOut.cs and CSharp\FF_SpeechSynthesis.cs\VoiceSynthesis.cs but they don't seem to play nice with Unity.

Lance_R_ · ‎06-14-2015

Fixed your problem, Joe! I've adjusted three things. One, be sure to clean up after yourself by using the Dispose() methods on certain objects. Two, you should use the SenseManager prefab in Unity instead of individual sessions. Three, the overloaded AudioClip method "public static AudioClip Create(string name, int lengthSamples, int channels, int frequency, bool _3D, bool stream);" in Unity 5 that you are using is deprecated and will break on dynamically created clips as described here:

http://docs.unity3d.com/Manual/UpgradeGuide5-Audio.html

...because the _3D flag is no longer a property. Easy fix though. Just remove the flag parameter. Here's the working code that doesn't crash on my computer.

using System;
using System.Runtime.InteropServices;
using UnityEngine;
using System.Collections;

public class SpeechTesting : MonoBehaviour {

    private PXCMSenseManager _pxcmSenseManager;

    private static PXCMSpeechSynthesis tts;
    private static AudioClip clip;
    private new static PXCMAudio audio;


	// Use this for initialization
    void Start()
    {
        _pxcmSenseManager = PXCMSenseManager.CreateInstance();

        if (_pxcmSenseManager == null)
        {
            Debug.LogError("SenseManager Initialization Failed");
        }
        else
        {
            int nbuffers;
            int nsamples;
            AudioSource aud = GetComponent<AudioSource>();
            String name = "test";
            Int32 sid = 1;

            _pxcmSenseManager.session.CreateImpl<PXCMSpeechSynthesis>(out tts);
            PXCMSpeechSynthesis.ProfileInfo pinfo;
            tts.QueryProfile(0, out pinfo);
            pinfo.language = PXCMSpeechSynthesis.LanguageType.LANGUAGE_US_ENGLISH;
            tts.SetProfile(pinfo);
            tts.BuildSentence(1, "This is finally working!");

            nsamples = tts.QuerySampleNum(sid);

            if (nsamples != 0)
            {
                tts.QueryProfile(out pinfo);
                clip = AudioClip.Create(
                    name,
                    nsamples * pinfo.outputs.nchannels,
                    pinfo.outputs.nchannels,
                    pinfo.outputs.sampleRate,
                    false
                );

                nbuffers = tts.QueryBufferNum(sid);

                for (int i = 0, offset = 0; i < nbuffers; i++)
                {
                    PXCMAudio audio = tts.QueryBuffer(sid, i);
                    PXCMAudio.AudioData data;
                    pxcmStatus sts = audio.AcquireAccess(
                        PXCMAudio.Access.ACCESS_READ,
                        PXCMAudio.AudioFormat.AUDIO_FORMAT_IEEE_FLOAT,
                        out data
                    );

                    if (sts < pxcmStatus.PXCM_STATUS_NO_ERROR) break;

                    float[] samples = data.ToFloatArray();

                    clip.SetData(data.ToFloatArray(), offset);

                    offset += samples.Length;

                    audio.ReleaseAccess(data);
                }

                aud.clip = clip;
                aud.Play();
            }
        }

        tts.Dispose();
        //_pxcmSenseManager.Dispose();  // actually it isn't necessary to clean this up, just the SpeechSynthesis object
    }
	
}

Joe_C_3 · ‎06-14-2015

Thanks Lance - I was actually able to get this running on Friday by using threading and using Dispose in the correct places. However I didn't know about the 3d flag and missed PXCMSenseManager completely so I'm really happy you posted these fixes (strangely my code was not crashing without the fixes but this is a good fix regardless).

Now I'm facing two totally different issues

I want to customize the voices and languages but can't seem to find a way to do it. I'm not sure if I need to install some nuance sdk library or if this is possible through RealSense. It does seem that its possible to install new voices downloaded from Nuance, but I need to create my own voice and language packs. I can't find documentation on this or information on whether or not this is even possible
The included voices are somewhat disappointing. The expressive sentences seem to override the pitch settings. example: set pitch to 50 (low), and then play the sentence "Oh my god! What just happened to my voice?" the "oh my god" will be a higher pitch (around 120 - sound female) and the rest of the sentence "What just happened to my voice?" will be the low 50 pitch. It sounds pretty horrible and I can't seem to find a way to switch the expressive tones off (not possible perhaps) so i seem to be stuck to a single pitch if I want my program to sound closer to natural.

Lance_R_ · ‎06-14-2015

You're right about the threading, Joe. Although my code works fine most of the time, Unity seems to crash occasionally with a threading error.

Unity 5 RealSense SDK voice synthesiser crashes