Tag: MFA

Blog Entry – Research and Project Updates 5

Posted on by Matthew

After months of working on Permadeath the opera, I was given the opportunity to work backstage during the performance. Prior to this, I had never worked in a theater. I signed on as manager of the real time facial animation system, Faceware live. Luckily, the system setup, that I originally help create, worked flawlessly. The system setup was:

  • 2 computers – 1 for each actor
    • 1 computer is for one actor which is passing the data over to the other computer
  • 2 cameras
  • Area lights
  • Unreal Engine 4
  • Faceware Live server

I had one day of training at the last day of rehearsals. Nothing was too different than what I remembered. There were only a few tweaks that the director had updated. Those updates solved most of the issues it originally had. So what did I have to do? I watched over the actors’ faces in the software. Sometimes, the face tracking would get lost and break. One actor’s nose would lose tracking. The other actor’s lips would lose tracking. I had to make sure everything was set before the CGI was called to be played. The actors wiggled or moved around to fix the issues. Besides those minor hiccups, the CGI ran smoothly. We did see one issue where the character, Apollo, mouth was weirdly flapping around. The next day, we suggested a different way to act that scene out and it was fixed. From what I was told, no one saw the issue. Only two of us that worked on the facial animations saw the problem. I say that is a win. The experience was great. I got to meet and work with many talented individuals. Something I would not have had the chance to do.

 

Project update

I got in touch with the developer who makes the custom wrapper (https://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/) . He kindly sent me the project assets to study and work with. So far, I had little time to dissect the inner workings but I am excited to get the chance too. At first, I couldn’t get the project examples to work but it was simply because I imported the original Kinect plugins. I created a new project and imported the wrapper and it worked. Now what I have to do is create a guideline or create sprints. Without having goals laid out, I feel it is hard to progress. The Kinect has so many things it can do! Once I get the ball rolling, I believe, the project will start to move forward.

 

 

Blog Entry – Research and Project Updates 4

Posted on by Matthew

As I progress with my research with using the Kinect in Unity. I come to realize how difficult it is to use deprecated devices. I struggle to find documentation on classes and setups.

I have found a few YouTube videos and some tutorials. The main issue is trying to understand each class and what they are doing and what are they are connected too. On the positive side, I did get the Kinect to register and run in Unity.

The Kinect fired up and worked with out a problem. One tutorial used the bodysourcemanger to detect and track a joint to make a 3d gameobject move. I am struggling with the face section, once again I am restricted to limited documentation, I did get a face to be detected.

This linked helped: https://social.msdn.microsoft.com/Forums/en-US/20257887-4c2e-42a9-be77-926d91fbdae3/face-expressions?forum=kinectv2sdk

I haven’t figured out how to pull a basic expression result out. I am going to have to dig into the plugin classes to figure stuff out. I did find documentation (https://ratemt.com/k2gpapi/annotated.html) but I believe it is for an examples that might use a custom wrapper (https://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/) . I am going to email the developer about it. If it is OK, I will research the examples and try to understand the basics before I start diving into the code.

Unfortunately, I will have limited time this week for research as I will be working a the Permadeath Opera. As the Techinical Assistant Director of CGI, I am over seeing the Faceware software during the performance. If anything goes wrong, I will be there to react and fix the issue. Prior to this, I was working on some technical work. I have been working on fixing some skin weights for some characters. One character , Aphrodite, has feathers on her arms as a shirt.

Aphrodite skin was poking through when moving around. I fixed this by simply adjusting some skin weights. Unfortunately, I could do so much since it was hundreds of planes so I tweaked her skin texture. I took the planes and baked it onto her skin texture. From there, I blurred the feathers to make an average color. Now, if there is some “skin” mesh poking through, it will be invisible to the viewer. Another character I adjusted the skin weights was Apollo.

Apollo had some weird skin weights in his face. He was the first Faceware rigged character so it is to be expected he was going to look rough. At first it was an easy fix by pushing and pulling the weights. Ultimately, I had to adjust the transform and rotations of the joints in the Unreal Engine. It took a little more time than I thought but the results show it was the right decision:

The last character I had to tweak was Adonis. The technical director decided the cape needed to be re-worked. Due to time limitations, the cape simulations were scraped for a shorter cape that was going to be bound to the skin. I modified the model to spec and worked on the binding. The bind was a little tricky as the cape supports on his shoulders float. If the characters shoulder joints move, it reacts and moves.

Next week, I will be spending more time researching and developing the Kinect in Unity. For the time being, I will be deciding what should be prioritized and what should be my starting point.

Blog Entry – Research and Project Updates 3

Posted on by Matthew

This week of research has been held up and on delay because of a delivery issue. I purchased a Kinect v2 that has all the accessories required to run on the PC. Unfortunately, it was shipped across to the other side of the country. As I write this, I should be receiving the items tomorrow, 9/19/18.

Nevertheless, I decided to create a Unity project and started importing the required plugins. I also installed the Kinect SDK for Windows 10. I got it all imported and installed without a hitch.

I did see an issue about a script called joints as Unity has a script the same name. It shouldn’t be an issue to update the scripts and the calls. Unity did want to update the plugin scripts using deprecated calls. I let Unity do its thing. Hopefully, I don’t regret doing that. I opened an example scene and ran it. Of course, it ran but it didn’t throw an error! Be that good or bad news, who knows until I plug the device in.

FYI – anyone that has Kinect running on a computer or laptop that switches gpus, by default when rebooting/starting Windows 10, it will always switch on the dedicated GPU. Without the NVIDIA notification icon, I would be wondering why my fans are constantly running and battery life that is trash. Easy fix is to end the tasks in the task manager. I haven’t found a way to stop it from being started by default. I will have to look at it some more but I fear if I mess with, I will cause more harm then good at this point. So I am going to hold off until I can test the Kinect.

I am excited to play around with the Kinect. Not only because of the project but also to expand the use of the device. I am curious if I can create animations using it as well as scan objects. If so, I want to use some of the features to help build the art around my project to help speed up the production.

Barber Shop Video

I remember something like this video at Disney World Hollywood Studios. It was called “Sounds Dangerous! Starring Drew Carey”. It was closed in 2012 and replaced by numerous different shows but the audio show had the same idea about the binaural audio. It used Dolby as the audio driver. Essentially, you watch a video while having a headset on. The lights would turn off and the audio show begins. There is a part where Drew Carey goes to the barber and he gets a haircut. The video felt and sounded the same as I remember the show did.

In the video where they put a bag over the microphones, I felt like there was a bag on my head. I started to panic and opened my eyes. I am surprised how well the illusion works. The buzz cutters sound sounds life like as well. I am curious how many games and movies use this audio technique. I feel like it would create a better immersion into their worlds. Would it work with a surround sound system or only with headphones?

 

Meow Wolf

Meow Wolf is an interesting place. At first, it looked underground and run down but now they have expanded so much! It is quite impressive. I like the idea where they expand to other cities, and have local artist work on pieces of art to display in their “museum”. I use “museum” loosely.

It looks more like a take on a modern “museum”, where the rooms are the pieces with pieces within the room’s world. One piece that was shown was the black and white room. I would have liked to see more of that piece. It reminded me of some comic book or something Tim Burton would make. Another thing that caught my eye was the “doors” to the other rooms. One was a fireplace where you would walk through. Another was a hole in the wall where you would crawl through. I am assuming there are regular doors but experiencing these “gateways” would help create the illusion. The illusion that you are in the art piece. Someday, I would like to visit one of these places. Not only for the experience but to be inspired by these “rooms”.

Blog Entry – Research and Project Updates 2

Posted on by Matthew

Setup Layout: (possible)

My idea for implantation for the Kinect in my project:

  • Audience / viewer approaches or walks by
  • Kinect detects and pulls data into Unity
  • Unity scripts will separate video and audio data
    • Video data will be face, color, depth, skeleton and # id
    • Audio data will listen for keywords and direction
  • Data will be push to a preset list of options
  • Options will be prioritized and sorted then picked upon
  • The combined option data will be passed to an animation to play that animation/audio
  • Once animation is completed return to the begin or have the new segment cached waiting until animation is done to play the new animation sequence

“Kinect is the eyes and ears of the piece.”

                  These features of the Kinect are important because it the eyes and ears of the piece. Without the Kinect, the piece will be static and non-interactive. It won’t have the life and emotion that I am striving for. For example, facial recognition will help the piece because I plan to use it to receive important data points.

One way facial recognition data would be useful is by determine if the audience or viewer is focused on the piece. Another would be by determining what facial expression they are making. I could use that data to point to and select the best animation sequence. Another way I could use facial recognition would be to determine when to start listening for audio input. I could use facial recognition to help determine gender if I needed too.

Color data is another useful input for the piece because I could try to determine the color of the viewers clothing, eye colors, and so on. I could even capture the image to save it and display it on the display if I wanted to add another feature. (< possible addon)  Another data capture would be interesting is by using the skeleton tracking.

Skeleton tracking could be used as a confirmation if the user is interacting with the piece or not. It may also be used to determine how far the audience is from the piece. I could try to call over the audience if it is determined they are too far from the piece. If it works, it will help the facial recognition feature by moving the viewer closer in. That way, I can receive better and more accurate results to process. I may have to combine it with id tracking but, from my understanding, I could also determine how many people are standing in front of the piece. I think the total can only be 6.

With all these features, I believe the piece will come alive. These fine details, when combined, should “wow” the audience. Either, they will be excited or creeped out by the piece. Either way, they will have an experience! The next step in the journey is testing the Kinect and getting the plugin to work to within Unity as the plugin is quite old. Nevertheless, I am excited to work the Kinect.