Research & Project Updates – 2020 #13

This week I was under the weather for a few days but got a bunch done regardless. On the character creation class here is the final link: http://www.mdhopkins.net/virus-ara-prototype/

On my thesis project, I finished my to do list. I adjusted the characters arm for the normal pose and sad pose. I had issue with the user voice search. It ended up being the index changing in the list. I swapped it out for a dictionary and reworked the code to get it to work. Another issue, was when it had to pick another story to tell, it would cause and error. I needed to set the sentence ID to 0. The last major fix was the issue with the head rotation in late update. I updated the animations so there is blending between it all but the head would snap when locking onto the user. Found help on the Unity forums. I had to create another rotation var in the start() and then call it again. All the original code worked as well. As of now, I need to do some long run testings aka over night testing for idle to see if it breaks. Hopefully it all works as intended. After that I am not sure what is left, I could do more voices with different emotions for the different moods but I am not sure how the writting would be. To be honest, I am gatekeeping this project because it is on the home stretch now…

Research & Project Updates – 2020 #11

The stuff I worked on this week: Character creation, Thesis project, and personal project.

Character Creation, I messed with textures and made sure the hi res was done. Here is an image:

Thesis Project:

I fixed some bugs during test. I am still polishing but also working on the audio and lipsync. I realize how difficult it is to blend the sound to make it feel right. I am still not sure of my voice with it but I think it adds to the piece.

Personal Project:

I worked on a Winchester Rifle for fun. I was hoping to finish most of it in a weekend but I had to put it on the side for a bit. I returned to it and did a quick texture test and bake. There are some issues still but its not bad! As it stands, it is about 3,600k polys. I will come back to it and finish it up once I get time. Most likely after graduation.

Research & Project Updates – 2020 #10

This week I worked on the character creation class. I am at the point of finishing up the detail work in zbrush and bringing it into substance Painter. As for the project, I fell behind and did not get the audio recorded yet but all the code is working. Just need to add in the content. On top of it, I finished the family canvas image and added into the project. One less thing done. Here are some of the weeks work screenshots:

 

Quick testing: more to come –

Research & Project Updates – 2020 #9

This week I finished up the character retopo and uving. I made sure I brought it into zBrush and re projected. I am looking forward to texturing the creature.


For my thesis project, the major struggle to finish the code is now done. The only real test left for it is to use the kinect and see if there are any bugs. I am guessing the only thing that could would be the switch when it check if a person is present or not. The rest should work pretty good. I’m actually excited to move on and start doing the audio and animations. My mind is burnt from this rewrite… Here is all the functions that are required to make the magic happen.

Research & Project Updates – 2020 #3

This week I have worked on fixing some locking bugs. I got lipsync, audio, and animation to work with the storyID. I have a method developed and now have to implement it into the project. I am expecting a couple of hours to work it out. The major issue right now is locking the system so it does not loop but rather wait until it is finished animating.

 

Character Creation:

I worked on the base mesh. I am pretty happy with the overall character. I need to rework the torso and armor shapes but it is going in the right direction.

 

Thesis – Project Blog 13-14

Procedural Modeling

Me this weekend with Houdini:

Over the past few days, I have been hard at work finishing up my car generator. With the crashes with Houdini and Unreal, it was frustrating to say the least. Luckily, I swap over to Unity, and fix some code which saved time as it was not crashing as much. Besides the craziness, I believe the experience was worth it. I finished the project and got to learn Houdini in the process. Here are some Screenshots and Video:

Blog Entry – Research and Project Updates 7

This week’s research contains with speech recognition. I got the Kinect speech functionality to work in Unity to display my command and the percentage of confidence. I believe this is not going to work for my project as I need more functionality than only commands. The way it recognizes phrases does not seem to understand the phrase when it is used in a sentence. After some debate, I was recommended to try out a few different services that are tailored to my needs.  – https://blogs.unity3d.com/2016/08/02/speech-recognition-and-vr/

 

IBM Watsom:

https://www.ibm.com/watson/

https://github.com/watson-developer-cloud/unity-sdk

Google Cloud Speech:

https://cloud.google.com/speech-to-text/

https://assetstore.unity.com/packages/add-ons/machinelearning/google-cloud-speech-recognition-vr-ar-desktop-desktop-72625

 

Unity Built in – Windows 10 – UnityEngine.Windows.Speech

https://docs.unity3d.com/ScriptReference/Windows.Speech.DictationRecognizer.html

 

Plugin for Unity that contains all:

https://bitbucket.org/Unity-Technologies/speech-to-text

 

Only difference on paper between Google and IBM is the amount of free monthly send requests. Google is 60 minutes and IBM is 100 minutes. After the free monthly use, the prices are comparable. I do not think I will pass the limit but if I do, I will make sure there are measures in place. Side note – I believe the windows speech might do exactly what the Kinect was doing. I am going to be testing that one first as it is already built within Unity.

Next week sprint:

Try out, one or all, speech to text services within Unity and decide what service is the right one to use.


Side Project – Turn HopShock into a infinite auto jump art piece

I am working on transferring my game, HopShock, into a automated art piece. As of right now, I removed the GUI, and other interactivity. Only thing left to work on is automating the jump and having the character randomly get hit by the spinning circle. I am planning to work with my cousin, who was the lead programmer on the project, to help create a successful solution. I am expecting at the end of the week, the edits will be done and fully working on an iPad. If not, then by this weekend.

 

 

 

 

 

 

Blog Entry – Research and Project Updates 6

This week’s sprint required me to detect if a person is present in the scene, and to detect/track a face and the face features the Kinect sensor registers. I completed it using the  Kinect v2 Examples with MS-SDK and Nuitrack SDK  plugin for unity. There are two main necessary scripts to use. One is KinectManager.

KinectManager script is required for the Kinect to initialize and run. It also contains numerous classes that are important for body tracking and could be useful later in the project. The other script is called FaceTrackingManager. This script is like the KinectManager but only manages the face and head of the user. With these scripts, it is easy to debug any issues. For instance, I have it set up to create a small window with a color map video. In that video, right now, displays a rectangle box around my face. It enables me to see if the face tracking is working. With those two scripts working, I edited a script that was a demo. That script was only to detect a smile. I added in all the face properties Microsoft allows. I should be able to call the variables once I start creating my database. These properties should give a direction to what script/animation needs to be played at a given time.

Next week’s sprint:

Get speech recognition to work with some basic phrases. Then create a script to call all the input data and display them.

 

Also, as I mentioned above, the KinectManager has some useful classes so I wrote them down. Later, I discovered a website that has it all indexed:

https://ratemt.com/k2gpapi/annotated.html

 

Blog Entry – Research and Project Updates 5

After months of working on Permadeath the opera, I was given the opportunity to work backstage during the performance. Prior to this, I had never worked in a theater. I signed on as manager of the real time facial animation system, Faceware live. Luckily, the system setup, that I originally help create, worked flawlessly. The system setup was:

  • 2 computers – 1 for each actor
    • 1 computer is for one actor which is passing the data over to the other computer
  • 2 cameras
  • Area lights
  • Unreal Engine 4
  • Faceware Live server

I had one day of training at the last day of rehearsals. Nothing was too different than what I remembered. There were only a few tweaks that the director had updated. Those updates solved most of the issues it originally had. So what did I have to do? I watched over the actors’ faces in the software. Sometimes, the face tracking would get lost and break. One actor’s nose would lose tracking. The other actor’s lips would lose tracking. I had to make sure everything was set before the CGI was called to be played. The actors wiggled or moved around to fix the issues. Besides those minor hiccups, the CGI ran smoothly. We did see one issue where the character, Apollo, mouth was weirdly flapping around. The next day, we suggested a different way to act that scene out and it was fixed. From what I was told, no one saw the issue. Only two of us that worked on the facial animations saw the problem. I say that is a win. The experience was great. I got to meet and work with many talented individuals. Something I would not have had the chance to do.

 

Project update

I got in touch with the developer who makes the custom wrapper (https://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/) . He kindly sent me the project assets to study and work with. So far, I had little time to dissect the inner workings but I am excited to get the chance too. At first, I couldn’t get the project examples to work but it was simply because I imported the original Kinect plugins. I created a new project and imported the wrapper and it worked. Now what I have to do is create a guideline or create sprints. Without having goals laid out, I feel it is hard to progress. The Kinect has so many things it can do! Once I get the ball rolling, I believe, the project will start to move forward.

 

 

Blog Entry – Research and Project Updates 4

As I progress with my research with using the Kinect in Unity. I come to realize how difficult it is to use deprecated devices. I struggle to find documentation on classes and setups.

I have found a few YouTube videos and some tutorials. The main issue is trying to understand each class and what they are doing and what are they are connected too. On the positive side, I did get the Kinect to register and run in Unity.

The Kinect fired up and worked with out a problem. One tutorial used the bodysourcemanger to detect and track a joint to make a 3d gameobject move. I am struggling with the face section, once again I am restricted to limited documentation, I did get a face to be detected.

This linked helped: https://social.msdn.microsoft.com/Forums/en-US/20257887-4c2e-42a9-be77-926d91fbdae3/face-expressions?forum=kinectv2sdk

I haven’t figured out how to pull a basic expression result out. I am going to have to dig into the plugin classes to figure stuff out. I did find documentation (https://ratemt.com/k2gpapi/annotated.html) but I believe it is for an examples that might use a custom wrapper (https://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/) . I am going to email the developer about it. If it is OK, I will research the examples and try to understand the basics before I start diving into the code.

Unfortunately, I will have limited time this week for research as I will be working a the Permadeath Opera. As the Techinical Assistant Director of CGI, I am over seeing the Faceware software during the performance. If anything goes wrong, I will be there to react and fix the issue. Prior to this, I was working on some technical work. I have been working on fixing some skin weights for some characters. One character , Aphrodite, has feathers on her arms as a shirt.

Aphrodite skin was poking through when moving around. I fixed this by simply adjusting some skin weights. Unfortunately, I could do so much since it was hundreds of planes so I tweaked her skin texture. I took the planes and baked it onto her skin texture. From there, I blurred the feathers to make an average color. Now, if there is some “skin” mesh poking through, it will be invisible to the viewer. Another character I adjusted the skin weights was Apollo.

Apollo had some weird skin weights in his face. He was the first Faceware rigged character so it is to be expected he was going to look rough. At first it was an easy fix by pushing and pulling the weights. Ultimately, I had to adjust the transform and rotations of the joints in the Unreal Engine. It took a little more time than I thought but the results show it was the right decision:

The last character I had to tweak was Adonis. The technical director decided the cape needed to be re-worked. Due to time limitations, the cape simulations were scraped for a shorter cape that was going to be bound to the skin. I modified the model to spec and worked on the binding. The bind was a little tricky as the cape supports on his shoulders float. If the characters shoulder joints move, it reacts and moves.

Next week, I will be spending more time researching and developing the Kinect in Unity. For the time being, I will be deciding what should be prioritized and what should be my starting point.