12!! 💥

Maybe Maren can turn on recording as well? In case that I fail asleep at 1am or 2am?

12 submissions for the twelfth year as it’s growing linearly.

http://www.pcgworkshop.com/schedule.php

@Jialin I can only stop the recording, it seems we can't record both at the same time

Ok I see

I will take some tea later :)

The voice channel is on the same category as the pcg-2021 text channel

Where can we find the pdfs for this workshop?

Not all are online yet (I think), but this one is at https://arxiv.org/abs/2107.12524

Papers will be available here: http://www.pcgworkshop.com/database.php Apologies that we did not put them online earlier

Thanks

*claps*

Nicely done Bowei!

Excellent work!

Really interesting job, thank you for the presentation.

any workflow for generating these rooms with MdMC? I am not sure if i missed something, i do not see , how ensemble method works

doesn't splitting up the data horizontally and vertically reduce the amount data available for training and lead to worse models?

Thanks!

Maybe the two models can be trained simultaneously in some way so that their cooperation ability (decide which model to be used next to generate a room and the accessibility) can be evaluated as well?

Bowei: I'd recommend looking at maybe using a GraphRNN approach (pytorch has a graphRNN library now), so you can capture longer term inputs in a general graph structure.

thx!

Thank you Bowei!

Thanks for the comments and questions all. Notably, the important thing was not to create the perfect Mega Man level, but just to test Ensemble Learning.

Are you going to record all at once or one record for each presentation?

Sure! Great to see ensemble learning here!

Definitely agree that there are better models for Mega Man!

Can hear it!

I can hear it too!

So many awesome papers from University of Alberta. I think I should make you guys a visit soon.

Please do!

do it!

It is not that far from Winnipeg

This paper is here: https://arxiv.org/pdf/2107.12533.pdf

Are you from ASGAARD lab?

Nope, that’s Cor-Paul’s. I’m the advisor of this lab.

Can you share your lab’s webpage, please?

One doesn’t exist, sorry. Entirely my fault for not putting one together yet.

If yours is this one: http://guzdial.com/ I will reach out after the conference.

That’s my (out of date) website, yes!

@Matthew Are those papers based on UG students’ final year projects or the student projects in the module you teach?

Neither, just students who wanted to do research!

Creative and motivated students!

For sure! UofA students are fantastic.

Nice job Zisen!

Have you run any statistical tests to check if there is a statistically significant difference between the three approaches?

Did you run any kind of test, like and ANOVA test to see if the p-value is less then 0.05?

*an ANOVA

Thank you!

Thanks for listening!

Thank you for the presentation Zisen!

Thanks

Great presentation Zisen.

Thank you for the presentation!

We can still hear you

Can still see it ok

we can hear you

You’re a little robot-y, but it’s fine

There’s a bit lag, but its file

*fine

Now he's gone :)

It was getting too exciting :)

He created the hype and left, that is not fair, lol

the hype :D

Can we get a link to the paper posted in chat?

https://arxiv.org/abs/2107.02457

Merci

Great session so far. Sorry that I have to leave already. 👏

Thank you for the paper link, this work seems quite interesting!

Sounds sensible

That would work for me.

10:35

What time is it now for reference?

meeting break 10:35

10 minute break

👍

:)

Are you going to pause the recording?

Wellcome back JP

Merci beaucoup JP, great presentation. Your work is really interesting.

Really great presentation and interesting work Jean-Baptiste!

Reasonable for this kind of exploratory work

to just report p-values as is I think

(I agree the approach isn’t totally unreasonable, but I think multiple-comparisons corrections should be used whenever possible; it’s just something that not everyone knows to try)

I use the Bonferroni correction for t-tests, when doing multiple comparisons. How do you correct multiple-corrections for correlation critical values, Peter? Maybe post in the discord for persistence?

Sure, I’ll post on discord

Good idea Liapis

Thank you for the talk Jean-Baptiste!

Merci beaucoup Jean-Baptiste

The sound works well. Thank you!

Really cool stuff! Have you considered what skip-grams might look like in this context?

Thank Colan, it was a great presentation.

Thank you for the presentation Colan! Towards the beginning of the presentation, you mentioned that n-gram approaches are effective at capturing the "style" of a level / game: would you be able to elaborate on what you mean by style here?

I couldn't follow how truncation preserves the level quality? doesn't that break some n-grams?

That would be great, thanks for the explanation Colan!

n-grams harry potter: https://bi3mer.github.io/blog/post_22/index.html#year=2019

@Colan: If I got it right, you used some metrics like the number of spikes and other stuff to check if the level is difficult or not. Have you let real players play it and check if they agree with your measurements about how difficult the level was?

(Wave-function collapse is actually somewhat similar to multi-dimensional N-grams...)

@Rogerio, it was something we wanted to do and discussed but wasn’t really possible given the pandemic. We are planning to do something with real players down the line when we use the segments for future work

Awesome, thanks Colan. Great presentation by the way.

Screen froze for me, anyone else?

+1

Oh it came back

unfrozen now for me

it was like 3 seconds but is good now

Happy to see tools for scene integration, since that's a powerful part of the algorithm that has previously been somewhat intimidating for non-experts.

Where is the paper?

Very cool stuff! Thanks for the talk.

Really nice work!

Thanks for the talk!

Thanks for the great presentation!

Great talk Adam, thank you.

Have you considered interactive methods of adding constraints? (eg. the user being able to click on bad matches to add that as a negative constraint)

“Example-Based Model Synthesis”

https://paulmerrell.org//model_synthesis.pdf

The conference vs. GitHub factor is real I think!

Very gif-able as well.

I at least saw WFC outside of academia first

I'm also not a fan of the WFC name

well, Paul's work on MS was pretty much impactfull at its time...

Coffee break! :)

Are we back?

Patapoon….

A lot of metrics discussion at the workshop this year, interesting!

^+1Maybe it is time to blow the dust of the expressive range papers !

There’s been some pretty cool expressive range follow-ups I think! Mike Cook’s Danesh, and Adam Summerville’s use of e-distance to compare original data distributions and output content distributions.

Great job Emily!

Great presentation.

Thanks Matthew, I will look into them!

Thank you for the presentation Emily, nicely done!

Really interesting work, great to see work on rhythm game generation!

I agree Jim

I'd like to see even more expressive range followup, and Adam Summerville's "Expanding Expressive Range" should be part of that.

Thanks for the presentation!

I added expressive range and metrics to possibel discussion topics.

Good question Maren.

Maybe a next step would be let people play it and evaluate their feedback.

Can I have the link to the Discord server?

https://docs.google.com/document/d/1PpVsUcSmmfXgB4G9umC6FnDqj4IiEOYzRe2nS0CXtPM/edit?usp=sharing

Alberta is rocking this year!

@Arash the link should be in your email if you’re registered for FDG

https://arxiv.org/abs/2107.12506 (arxiv link to taikonation!)

One of my favorite Apple II games!

I should have linked that, sorry Emily!

Link to Kynan’s paper https://arxiv.org/abs/2107.12532

It seems like the player path would be highly dependent on the position of enemies... Is there a way to condition either the LSTM's generated path with enemy positions from MdMC, or influence the MdMC's enemy placement using the LSTM path? Also, what about placing the ladder that appears once the player collects all the gold?

Nice job Kynan!

Nice presentation, thank you Kynan.

Thanks Kynan, very interesting work and results

Thanks for the presentation!

Thanks for the presentation Kynan!

Thanks for the presentation!

I think we're due for a conversation about metrics.

I agree, I want to discuss metrics

Thanks, all presenters + organizers!

https://docs.google.com/document/d/1PpVsUcSmmfXgB4G9umC6FnDqj4IiEOYzRe2nS0CXtPM/edit?usp=sharing

I agree that any metrics that touch on affordances rather than relying directly on raw level (or other) data are usually good ones?

Adam Summerville's "Expanding Expressive Range" is another one.

I agree it is a chicken-egg problem.

Yeah, similarity or diversity metrics won’t be *directly* comparable between domains…

I feel it's also still difficult to figure out how to model more nebulous concepts like "This level is fun" or "This level looks nice" … is this even what we want to model?

But at least if two papers are both using versions of those we can get some indirect sense of performance?

In software engineering there is a lot of focus on metrics. When determining which metrics to use in a new area, a widely used approach is "Goal - Question - Metric" or "GQM" for short.

https://en.wikipedia.org/wiki/GQM

Thanks for the link Jim, that looks interesting!

The general idea is to start with high level goals on what you're trying to achieve, then translate that into more precise questions, and then metrics are used/developed to provide information which can answer those questions.

Or, another way to look at it is to do a kind of requirements phase first, then use that to guide metrics selection.

“Controllability” is how this problem has been talked about in some prior work

Or the one I heard about this week: is the note in the same font as is used by the hospital that we've seen a lot of cases from.

@Isaac I think I saw that article too :/

The commercial games industry used to use Metacritic scores as their measure of success

Regarding Playability as a metric: This assumes we have access to a human-like solver for all of our games though right?

@Maren yeah that can be a barrier in some domains, although often it doesn’t need to be very human-like in order to be successful

That’s Mario for many of us Adam.

A lot of machine learning advances are driven by relatively small improvements on standard datasets.

Ah, we can solve the metrics and the scicom problems with one stone by releasing our algorithms as games and evaluate them based on the Metacritic score :P

And people get annoyed about it (see Christoph’s joke-y comment about being glad Emily’s work wasn’t just platformers)

It seems like it would be hard to evaluate how well the Taiko drum model performs on mega man levels though :3

I do wonder if some of that is a bit of a problem though, being focussed on a specific dataset score maybe misses the point?

“Used to be” jeez Christoph

+1 Christoph

Can there be a metametric to evaluate metrics?

I've been working on Super Metroid recently, and adding different powerups that alter player movement abilities makes it hard to apply some of the Mario work

for example, a lot of papers on Mario use player position as a proxy for player state

It'd be nice to see more ablation studies in PCG

Mario Maker games often lean very heavily on the various powerup states to make the levels interesting

Expressive range analysis is only as good (or maybe not always even as good) as the metrics it uses as axes though…

@Phil yeah that’s a good idea, but of course one usually uses metrics to compare ablated versions to the original… :/

@Hendrik I’d check out Adam Summerville’s paper on e-distance (Expanding Expressive Range)

There's probably room for a survey comparing different existing algorithms against new metrics

Yeah, more survey papers maybe useful in the field?

Even just a working corpus of a lot of different generators would be useful, so we can compare with past work more easily

Isaac: yes, that would definitely help. Having a PCG model zoo :)

Gotta make our PCG systems gif-able!

I work in video generation, so yes youtube :)

Maybe the PCG workshop should add a SciCom element?

Every academic should be animators on youtube :)

A GDC session kind of already exists with Emily Short’s Experimental AI session (running for 2 years now)

hehe.. why not eh?

The problem with putting a lot of work into a video presentation: As a PhD student, you get nothing out of it as it doesn't really count towards your publications towards getting your PhD. At least for me it sadly doesn't count

There’s a lot of variety there, Ubisoft La Forge, EA’s Seeds, Unity ML, etc.

Reminder for those who pre-recorded their presentations: you can and should just make them public on YouTube and link to the paper in the description.

That way it’s no more work than the conference presentation.

+1 Matthew

Seconding Maren's point, as a field we need to reward Science Communication more.

+1 Matthew

+1 Matthew

Academia simply does not reward that work at all...

Maybe the metrics we should be talking about are the tenure committee metrics.

I think this is happening now Christoph. I don’t think we need to wait 20 years.

I've found recently that a lot of companies are definitely on board. But there's still a lot of inertia

@Peter, it depends where you’re at. I get points towards tenure for science communication.

It would have been probably 2-10x more work for me to create a really good explainer video, with animations etc., compared to the video I did produce for this conference...

I also think it's important to reach out and talk to people in industry about this. Those of us who have game degree programs have alumni networks we can reach out through to make these connections.

@Matthew that’s awesome!

Townscaper does some of that right?

A good chance to ask the questions in this panel session: https://aingames.cn/ieeecis-sa/

Panel session :"Academia or industry? Why I decided to work in game industry." By speakers from game industry

Awesome, thanks for sharing Jialin.

I will definitely ask the people from game industry about how they thing about PCG and related topics.

+1 on just putting the slides and voiceover out

Plus _you_ don't need to make all of the presentations.

As well, people like Luke Dicken, who leads a design-focused game AI team within Zynga see part of their role as bridging academia and industry. He'll be here for the main conference, and Zynga will also have a GatherTown room where you can talk to Luke and other Zynga folks (most of whom came from academia and have attended FDG before starting to work for Zynga).

Don't compare yourself to those top science explainers with millions of views

I'm from the games industry and became an academic. But I think Christoph is right, there's still a lot of inertia and buy-in for AI and PCG in general needed.

The transcript is somewhere in the zoom options

I think click the three dots button?

@Phil: what made you decide to became an academic, if I may ask?

Could one solution be to experiment more with mods and less with "homemade" solution? This could be a way to have more "appealing" content to showcase, while also sergin as a proof that modern game engine can run complex AI solution.

serving*

@Jean-Baptiste yeah that makes sense

At the same time, we want the game engine and data from industry :)

I’m NDAed or I would offer a few counter examples.

Yeah, indies are definitely on board

@Jialin I wish we had the data too... But regarding the game engines, a fair amount of games are built with Unreal Engine now

One related thing: the popularity of randomizers for old games like A Link to the Past and Super Metroid have influenced some new games coming out to add similar game modes

e.g. Bloodstained

ProcJam has a number of great Unity tutorials as well.

I'd second that, yes people want to drop in solutions first and experiment second

So clearly the solution is to force Mike Cook to make tutorials for all our approaches :)

The Games in AI YouTube channel that Tommy Thompson runs is another example of useful communication

Metrics are important, because we need something to measure and compare, but because we are talking about games, it is also important to have feedback from players to make sure we are delivering something fun to play. Talking about “quality” and “fun”, maybe it is possible to adapt the System Usability Scale (SUS) for games, and that could help to evaluate the results in the players’ perspective.

Is this at some point too much to ask of one person? Should we researchers also be the ones to make the YouTube videos or would it be better to work together with someone who does that professionally? Would that even work for academia?

I know that some grants include funding for science communication.

I'd recommend the paper Meta-Sim2 by someone from Sanja Fidlers lab and NVIDIA, where they use computer vision and RL to build procedural scenes for automated driving in Unreal Engine

Thanks for the great workshop organizers!

I think if all we want is to promote paper via youtube, researchers are capable enough to make a ~1min short video to just demonstrate quickly how the system works

and I find people have a tendency to click on short videos because they are not costing a lot of times

the pcgworkshop website still doesn't have a link to the schedule :3

That’s not true Ross.

I think it does?

i can see the schedule

Yeah, I see it :)

http://www.pcgworkshop.com/schedule.php

+1 on more demos :)

Tom Betts did some great talks

and Oscar the guy who wrote Townscaper?

Here's my river network generator; http://topoi.pooq.com/hendrik/rivers/

Thanks Hendrik

@Hendrik thanks for the link! I’m working on something similar right now so I’ll check that out

Thanks for the workshop everyone! It was a great experience

Thanks for organizing the workshop!

Huge kudos to the organizers. Super successful workshop!

Thank you all so much!

Thank you Christoph !

Thank you all!

Thanks everyone!

Thank you all!

Great workshop, thank you all.

Thanks to all the organizers!

It’s been such a great experience. Thank you!

Thanks all!

Thank you everyone! It has been a great experience!

Thanks to all the organizers!