PDA

View Full Version : Singing Pumpkin Lip-Sync



andyb
08-25-2010, 08:38 PM
Well this year I wanted one of those 6' singing pumpkins on the house. I spent a bunch of time researching how to do this, and I really didn't find everything I was looking for in one place.

Now you can find it here, because I'm going to save you the pain.

First, you need to build a scaffolding for the pumpkin. I made mine out of five 10' pieces of 1/2" PVC piping, eight 1/2" T connectors, four 1/2" + connectors, and four 1/2" elbows. I also purchased (at Lowes), a roll of plastic hardware cloth (24" x 25' x 1/2" squares). I also used garden twist ties on a spool (for mounting the cloth on the frame, but zip ties would work). Then I purchased some spray paint to darken it all up.

Cut all of the PVC pipe to 23" (you will have one extra 23" piece). Build a tic-tac-toe shape with a frame around it. It should be 6' square. Unroll the hardware cloth in three strips and mount it to the frame with the ties, enough to support the weight of the light strands you will be putting onto the frame for the face.

Paint the frame a suitably dark flat color to hide it. so it doesn't reflect when the lights come on.

Mount either rope light, or mini lights to the mesh cloth to make the pumpkin face. I made triangular eyes. The lower part of the eyes are one channel. The upper two edges of the triangles for the eyes are on a second channel. This allows the pumpkin to "blink" approximately every 4-5 seconds for 300 ms.

The "mouth" has several shapes, and a channel for each. An upper lip, bottom lower lip, lower lip (just slightly above the bottom), a middle lip, an O mouth, and a smaller o mouth with some corners showing (this is the U mouth described later).

The pumpkin can be controlled with eight channels. So how to animate it? Here's what I found that was interesting. The various facial poses shown on this site were my guide --

www.garycmartin.com/mouth_shapes.html

Using the various lip shapes, I mimic the facial features shown by Gary. I started off by taking my lyrics and picking a letter like "O" and marking all of them, then using Vixen to animate the "O" mouth. Big mistake, and really slow progress....

So I read a very interesting web page:

www.keithlango.com/tutorials/old/lipSync.htm

which explains the principles of lip sync animation. Less is more. Ok, but I still am a geeky guy, and I wanted something to take text, and convert it to phonemes (the mouth shapes). I love open source, so I wanted something free.

What I found was a cool program that takes .wav files of audio, and typed in lyrics, and helps you layout the phonemes. It's called Papagayo, and it's available here:

www.lostmarble.com/papagayo/index.shtml

I took my .mp3 files and used audacity to chop them down to verses, and convert them to .wav files. Each .wav verse is loaded into Papagayo (trust me it's easier to work with smaller sections at a time!) and animated.

Before using Papagayo, I used a Vixen add-in called Tap-Tempo to determine the timing of the song I was sequencing. For example, I chose "Thriller", and I used a timing of 64, giving 16 events per second. The 16 number is used to set the FPS rate in Papagayo.

Now, loading the first verse into Papagayo, I immediately set the frames per second (FPS) to 16, then paste in the lyrics of the first verse. Use the tutorials to learn how to manipulate the sentences, then the words, then the phonemes. Use the software to get the mouth animations to look just right. After you've done that, you are ready to "Export Voice" or just use the grid display in Papagayo to map your phonemes into Vixen.

The mouth shapes in Papagayo are; AI, E, etc, FV, L, MBP, O, rest, U, WQ. In Vixen, on my pumpkin face, I mapped this to; AI = Upper Lip + Bottom Lower Lip, E = Upper Lip + Lower Lip, etc = Upper Lip + Middle Lip, FV = Middle Lip, L = Upper Lip + Bottom Lower Lip (same as AI), MBP = Upper Lip, O = O mouth, rest = nothing or Middle Lip -- depending on how it looked, U = U mouth, and WQ = U mouth.

Now, pick the first word (phoneme) and find the corresponding audio portion in vixen. In my case, Papagayo time index 18 = 0:58.752 in Vixen. At this location in Vixen, I used the AI phoneme to turn on the upper and bottom lips (that's my mapping). Let's say that the next Papagayo transition phoneme is at time index 22, and it's "etc". This means that the AI phoneme runs from time index 18 to time index 21, or 4 frames. In Vixen, the upper and bottom lower channels are on from 0:58.752 through 0:58.944.

Rinse, repeat for the entire verse, then the entire song. Errors in timing will creep in, so you may need to re-align the locations of the transitions in the sequence, but the mouth shapes should be correct for the lyrics.

What we really need is an add-in that takes the export from Papagayo, asks for the channel mappings from the mouth shapes to the appropriate Vixen channels, and the time index in Vixen for time index 1 in Papagayo. Presto, chango, a tool for quickly (relatively speaking ;) doing lip-sync.

I hope this helps someone else, and I welcome any comments, or even better -- a faster way of doing this.

Andy

budude
08-25-2010, 09:18 PM
Wow - great work putting this all together! I keep saying every year I'm going to go big with Halloween as I do with Christmas but never quite get there... Add this stuff to the wireframes sold by Dougp and you're set!

andyb
08-25-2010, 09:41 PM
Wow, hadn't seen Doug's stuff prior to this. I wish I'd known before building my pumpkin. In a few days, I'll add screen shots and pictures, if it will help.

Andy

kychristmas
08-25-2010, 09:46 PM
Wow, hadn't seen Doug's stuff prior to this. I wish I'd known before building my pumpkin. In a few days, I'll add screen shots and pictures, if it will help.

Andy

Don't kick yourself. Doug just posted his stuff. I'm very excited about his stuff too. Your info makes it evern better.

plasmata
08-25-2010, 10:00 PM
Definitely add pictures. I'm sure many of us would be interested in seeing them.

mahoo313
09-02-2010, 09:57 AM
andyb - any updates. I would love to see pics!

Dougp
09-02-2010, 04:13 PM
If you are having trouble sequencing the mouths, I have one set to Thriller that I can sell you. I did not program it, but it's good and I have to pay the individual that did it.

It only uses 3 mouths. Send me a PM if you want it.

budude
10-03-2010, 02:05 PM
I've been playing a bit with Papagayo from andyb's suggestion. I can definitely see the advantage into cutting the song up into bits. It is no fun at all to move the phrases around after cutting-n-pasting in the lyrics to an entire song. I wish that was one thing they would fix but it doesn't look this is really actively developed anymore. It's open source but my programming skills aren't even close to being able to make changes.

Anyway - I looked closely at my pumpkin and laid out where the various mouth breaks are so I would know how to run the lights. As this is a relatively small pumpkin, some of the mouth sections are only a few inches long. I bought some rope light for this with 18" sections so it looks like I'll have lots of black tape to hide parts of the sections. If I get it done, I'll post some pics.

I also laid things out in Visio and exported it to a PDF so you can match the mouths from Papagayo to the pumpkin. It's not exact but it will probably look just fine (or should I say - close enough...

Dougp
10-03-2010, 03:46 PM
Look's great. If you all see something that I need to do in making these to make it easier for you to put lights on them, please let me know. Send an email to info@wireframedlites.com

Doug


I've been playing a bit with Papagayo from andyb's suggestion. I can definitely see the advantage into cutting the song up into bits. It is no fun at all to move the phrases around after cutting-n-pasting in the lyrics to an entire song. I wish that was one thing they would fix but it doesn't look this is really actively developed anymore. It's open source but my programming skills aren't even close to being able to make changes.

Anyway - I looked closely at my pumpkin and laid out where the various mouth breaks are so I would know how to run the lights. As this is a relatively small pumpkin, some of the mouth sections are only a few inches long. I bought some rope light for this with 18" sections so it looks like I'll have lots of black tape to hide parts of the sections. If I get it done, I'll post some pics.

I also laid things out in Visio and exported it to a PDF so you can match the mouths from Papagayo to the pumpkin. It's not exact but it will probably look just fine (or should I say - close enough...

jrock64
10-03-2010, 07:16 PM
Anyway - I looked closely at my pumpkin and laid out where the various mouth breaks are so I would know how to run the lights. As this is a relatively small pumpkin, some of the mouth sections are only a few inches long. I bought some rope light for this with 18" sections so it looks like I'll have lots of black tape to hide parts of the sections. If I get it done, I'll post some pics.

never used it, but 12v rope light usually lists a 3 or 4 inch cut.

Joel

budude
10-03-2010, 08:54 PM
I have 120v rope light and it's actually 36", not 18" so I'm really going to have some extra... oh well that's why we have duck tape!

Rainlover
10-03-2010, 09:14 PM
Here is a similar idea I finished a little while ago. It uses 12 volt leds and 25 channels. The video is not that great, but you can get the idea.
http://www.vimeo.com/15112788

John

budude
10-04-2010, 12:59 PM
I've given up on using 120vac rope light for this, the lengths are going to be too messy so I'll be ordering some 12v stuff which will be safer anyway...

Aussie Ryan
10-05-2010, 04:45 PM
I take it you have seen this? http://www.youtube.com/watch?v=ARwl8qWoGrQ

budude
10-05-2010, 04:55 PM
Nope - haven't seen that particular one but many others - pretty good with the multiple lip-syncs on it.

dirknerkle
10-05-2010, 05:02 PM
I've given up on using 120vac rope light for this, the lengths are going to be too messy so I'll be ordering some 12v stuff which will be safer anyway...

Since when did you go for safer? I thought you were the kind of guy who lived on the edge!

:lol:

plasmata
10-06-2010, 11:37 PM
Brian, how many channels will one "face" take? I count 11. I really like all of the mouth shapes but I would make one small change to the eyes (if I'm interpreting your design correctly). I think it would be more natural for the closed eye to use the bottom line instead of the top.

Keep up the good work and keep us posted on your progress!

budude
10-07-2010, 03:19 AM
Brian, how many channels will one "face" take? I count 11. I really like all of the mouth shapes but I would make one small change to the eyes (if I'm interpreting your design correctly). I think it would be more natural for the closed eye to use the bottom line instead of the top.

Keep up the good work and keep us posted on your progress!

Yeah - there are 11 for every individual piece. The alternative is to wrap each mouth shape separately which you might get away with a larger pumpkin frame but it might get a bit bulky with the smaller one. When I start wrapping it might end up to be some hybrid .

I made a trip to BigLots! today and they had 120v rope light for $8 which is pretty cheap - so - so much for 12v - got all the colors I wanted (Red, Blue, Green and Orange) too. If I had more time/patiience I would get all the 12v stuff but this will work for now - just have to cover up all the extra. No one will see it at night anyway. I'm hoping to have this sucker running this weekend. I've got Thriller all ready and This is Halloween is close.

As for the eyes - - I like them with the angled look - it looks a bit creepier that way but it's pretty easy to make it either way. Anyway, I highly recommend Dougp's wireframes - they're very well made!

dirknerkle
10-07-2010, 10:13 AM
Brian, how in the world do you find time to do these projects?!?!?!? Last time I checked, we still had only 24 hours in a day here in Minnesota. Did the Governator change your time or something???

budude
10-07-2010, 08:24 PM
Brian, how in the world do you find time to do these projects?!?!?!? Last time I checked, we still had only 24 hours in a day here in Minnesota. Did the Governator change your time or something???

LOL - and this coming from the guy that's got about 500 hours of sequencing done - months ago....... Actually some stuff (strobes, etc) dropped off so my xmas show will be much like last years for better or worse so I decided to put a bit more effort into Halloween this year.

plasmata
10-07-2010, 09:28 PM
As for the eyes - - I like them with the angled look - it looks a bit creepier that way
I had a feeling that was the reason you did it that way. Its all a matter of taste.

I too am surprised how quickly your putting this together. (Easy for me to say looking from the outside. Not so sure you think so. :D ) Unfortunately my free time has been pretty limited this year and I probably won't get a display up. At best I might borrow one of the posted sequences and have a limited display. At least I'll have another year to "accumulate" lights for next year's display.

andyb
09-09-2011, 12:10 PM
Sorry everyone -- I got real busy last year after making the original post with work. Never had the time to come back and update this -- my bad. Here are some pictures of the pumpkin that I build using PVC frame, and plastic chicken wire. I also included a picture showing the lights strung up to the grid. I used black thread to tie the lights into the shape I was looking for.

If anyone is interested in the .vix file, I have a 24 channel (plus two beat channels) of MJ's Thriller (with Vincent at the end). The lip sync is two channels for the eyes (so he can blink), and six for the mouth.

This year I'm trying to finish Somebody's Watching Me. This new version of Vixen looks like it has the promise that I'll be able to integrate the lip sync software output into the sequencing...

AndyB

zztopmo
09-27-2011, 12:08 AM
I would be interested in the vix file. I have wondered about how to do a pumpkin this year and you answered my delima. Keep up the great work.

Rick

Livermore-Dad
09-27-2011, 12:47 AM
ditto.. :)

Tory

Wildbill
09-27-2011, 08:25 PM
Doubble Ditto :cool:

steve_hirst
09-27-2011, 09:53 PM
you would have to get with David Moore about the vix files as I think he doesn't want them distributed.

Wayne J
09-27-2011, 11:16 PM
I take it you have seen this? http://www.youtube.com/watch?v=ARwl8qWoGrQ

That is so cool! I would enjoy a smaller version of that.

kychristmas
09-27-2011, 11:28 PM
I believe this was created from Scratch using the tools described in the original post. It doesn't anything to do with David.


you would have to get with David Moore about the vix files as I think he doesn't want them distributed.

MacabreRob
09-28-2011, 01:25 AM
I have a 12-channel one you all can use if you like. I use 7 channels for the face, 2 for side ghosts and the remaining three for JOLs.

11547

Let me know what you think!

Livermore-Dad
09-28-2011, 12:22 PM
Thanks will give it a look :)

andyb
09-28-2011, 03:04 PM
Here is the version I created last year. It uses a version of Thriller w/ Vincent at the end that is 5 minutes, 58 seconds long (MD5 of MP3 is cf977afaa690877cdf83a8e1aba5602f). I've squished the profile into this version. Thriller_VP.zip (Unfortunately it was over 1MB, so I had to ZIP it).

ChiefWarrant
09-28-2011, 05:33 PM
I take it you have seen this? http://www.youtube.com/watch?v=ARwl8qWoGrQ

Does anyone have this that they are willing to share?

Thanks

Tony

steve_hirst
09-28-2011, 06:12 PM
I believe this was created from Scratch using the tools described in the original post. It doesn't anything to do with David.

sorry, I was in the wrong pumpkin tread.

barbotte
10-10-2011, 04:14 PM
any one has other halowenn song they dont mind sharing ... i already have thriller ,,, need 1 or 2 more
thanks

Wildbill
10-10-2011, 11:31 PM
pm me your email addy

Daryl1
05-31-2012, 10:48 PM
okay, is everybody kidding or is there something in Papagayo that I am missing? have tried loading one verse, two words, up to the entire song and all lyrics. once I get it all loaded all the text goes in the first 2 seconds of the song and I for the life of me cannot figure out how to change its position. I'm starting to think it would be quicker to just teach pumpkins to sing!

timon
06-01-2012, 04:58 PM
I've found that Papagayo seems to use so many sounds that when your try to map them it's just too many mouth changes to look right even when using the the program a user posted, sorry can't remember the name and I don't have my windows system up, to translate. What I'm doing is to use Papagayo to break the sound down then using that as a basic guide but then pick the shapes that best work with the props.

I've got a couple of the songs from HolidayCoro and you can tell which have been run through the translation and which ones have been done by hand. The ones done by hand just seem to be smoother.

That said I'm finding that figuring out which shapes to use when doing it by hand get easier the more you do. I really think after going through a couple of songs it will be quicker and easier by hand than modifying what comes out of the Papagyo.

John

Daryl1
06-01-2012, 09:20 PM
Thank you, I have about 20 hours trying to make it work and its not going well to say the least, I have finally decided to use audacity and a printed set of lyrics to get a voice track that I can read. and then replace it with the actual song, this seems to go way faster. at least for me it does. Just glad to see that I'm not going to have to teach pumpkins to sing. (i've heard they tend to not cooperate) :)

hollandlihjtdisplay
06-28-2012, 06:02 PM
Anyone have a four pumpkin face sequence to thriller they wouldnt mind sharing?

SLSettles
09-05-2012, 11:27 AM
I did a 5' lighted pumpkin display last season. I tried using Papagayo initially, but found it to be very cumbersome. I ended up just doing the old animator's trick of using a mirror to see how I form the words. I think an even better method would be to videotape yourself lip synching to the song and then step through the video, matching mouth shapes to time lines. Anyway, I thought I did a passable job for my first actual sequences.

http://www.youtube.com/watch?v=sTrDbDJmrUs&feature=youtube_gdata_player
http://www.youtube.com/watch?v=YhbNEzRte3s&feature=youtube_gdata_player

You might also note that the pumpkin itself is a cutout sheet (PVC sign stock in my case) and it is airbrushed with texture, shadows and ribs so that the prop looks good during the day... my wife would not have been happy with a black wireframe dripping with electrical cords.

lightman
09-05-2012, 11:49 AM
Nice job! I like the smooth lines for your mouth outlines. Gives it a friendlier feel.

The method of video taping yourself then following along to animate mouth movements is typically called rotoscoping. I used it to animate a volleyball talking to the team using stop motion animation.

I would superimpose the video of me over the volleyball, then select the phonetic shape, paste it on the volleyball, snap a picture, then repeat.

Took a long time, but the result was cool. But I digress..sorry about that.

Congrats again,
Lightman

somtng4u2c
09-05-2012, 10:19 PM
15205

Xenia
09-05-2012, 11:15 PM
Very nice. I like how you use the upper lip/middle lip combination for a closed mouth.

SLSettles
09-06-2012, 08:27 AM
Nice job! I like the smooth lines for your mouth outlines. Gives it a friendlier feel.

Thanks. Yeah, "friendly" was real important for me since all the kids know this is Santa Claus' house (I'm a professional Santa) so last thing I wanted was something to frighten the children. I probably redesigned the face three times before my wife proclaimed it Disney-enough!

On another front, wouldn't you know that within an hour of declaring Papagayo to be of dubious value I find that someone has made a "translator" that makes it enormously useful! Bob-O from the Light-O-Rama forums has created an app to take the Papagayo exported data and convert it into a LOR clipboard which can then be pasted into your sequence onto up to 10 consecutive channels that make up your mouth shapes. It works like a charm! I just wish there were a way to have it automatically create ease-in and ease-out when turning on channels which helps make the animation look more fluid. Here's the link: http://forums.lightorama.com/index.php?/topic/14473-voice-sync-software

Now, like I mentioned before, this app makes a LOR clipboard so it's probably no help to Vixen folks, but the thread speaks extensively about what it took to do the translation and I'm betting there's somebody here who could make one for Vixen.

Note that when you go to his software site to download the app and other bits, it's not entirely clear but to download the translator itself you click on the large parrot icon to the left. Note also that Papagayo does not handle large audio files well, so I suggest you break it into @30 second chunks (breaking at choruses or instrumentals). While you are editing audio, you might as well adjust the eq to maximize the clarity of the voice you are mapping just to make your task easier. Finally, you must set Papagayo to 25 FPS for the sync to stay locked in, it is set to 24 FPS by default.

I plan to sync several more songs this year and expect things to go much quicker thanks to this new Papagayo translator. Hope some others find it useful as well.

callen360
03-21-2013, 11:17 AM
Thanks for the info.

Crog
11-03-2015, 11:59 AM
Hello guys, just ordered 32 channew starter package from wowlights and am excited to try and build these pumpkin faces. Can someone tell me if it's better to use string lights or rope lights. A step by step tutorial would be nice as well. Looks like I'm going to try the pvc method.