Technical Sharing: How I make a photo-based AR game
Hello everyone. Finally I've finished my mid-term exam so I have some time to explain how I use reallife photos as game scene.
Actually, I was inspired by FF7. That famous game also uses a prerendered photo as the game scene and only draw the animated characters realtime. That makes cinematic scenes real on PS1.
So I decided to adopt this technique. There is a trade-off. The photo will makes my game more like a amateur game. But I think the innovation worth that defect.
First of all, welcome to play my game:
* https://ldjam.com/events/ludum-dare/38/clean-the-dorm *
Rendering Technique
In unity3d One can decide the order object was drawn with the * Render Queue * property of one material. In may project, the Render order is
Step One
First we draw the photo as the background image using a unlit shader with a camera space canvas with ZWrite on.
Step Two
Obstacle phase. What is an obstacle? In the scene view I turned on wireframe mode. We can see a lot of transparent boxes. They are the simplified geometry of the reallife scene.
The Obstacles plays 3 roles:
- Colliders in the gameplay
- Cut the virtual objects out if they should be obscured by the reallife objects.
- Receive the shadows of virtual objects and blend them onth the reallife photo.
I should emphasize the importance of shadows. Not because they increase the sence of reality. The Irreplaceable role shadow takes place is that they indicates the spacial relationship of objects. Without shadow one cannot tell whether the character in just on ground or floating in air. They cannot tell if it is a big character near the camera or a small character far from the camera.
Let's draw the obstacles. we keep ZWrite on, writing the depth of all the obstacles to block out the virtual objects to be drawn. We donnot use opaque mode to override the background photo by litted color. We use transparent mode, calculate the shadow intensity of the fragment and alpha-blend it onto the photo.
Step Three
Draw the virtual objects. I means the characters, the monsters, the particle fxs... Since the depth buff is already set by the obstacles, when the character should be occluded out by reallife objects, they cannot pass the depth-test.
Step Four
Ofcourse it's for ui.
Shaders for obstacle
```
Shader "FX/Cutout" {
Properties {
_Color ("Shadow Color", Color) = (1,1,1,1)
_ShadowInt ("Shadow Intensity", Range(0,1)) = 1.0
_Cutoff ("Alpha cutoff", Range(0,1)) = 0.5
}
SubShader {
Tags {
"Queue"="Geometry-20"
"IgnoreProjector"="True"
"RenderType"="TransparentCutout"
}
LOD 200
ZWrite on
Blend zero SrcColor
CGPROGRAM
pragma surface surf ShadowOnly alphatest:_Cutoff
fixed4 _Color;
float _ShadowInt;
struct Input {
float2 uv_MainTex;
};
inline fixed4 LightingShadowOnly (SurfaceOutput s, fixed3 lightDir, fixed atten)
{
fixed4 c;
c.rgb = lerp(s.Albedo, float3(1.0,1.0,1.0), atten);
c.a = 1.0-atten;
return c;
}
void surf (Input IN, inout SurfaceOutput o) {
o.Albedo = lerp(float3(1.0,1.0,1.0), _Color.rgb, _ShadowInt);
o.Alpha = 1.0;
}
ENDCG
}
Fallback "Transparent/Cutout/VertexLit"
}
```
* Important, You need to rewrite a new unlit shader in order to customize the Render Queue. The default unlit shader template is just ok. *
TODO find the original post I find the shader and make credits
Scene Setup Workflow
Camera Alignment
How to align the camera with the photo? Because the time limit I didnot try to challenge the maths monsters of computer vision. Instead I align the scene by myself.
First I googled my camera's field of view. There is something subtle: The field of view of reallife cameras are measured by the diagonal line of the sensor versus focal length. However in computer graphics the fov means the vertical height of the sensor versus focal length. It needed some calculation and I found my iphone6's fov is 34.55 Deg when it takes videos.
Later I found that several degrees of error in fov doesn't matter.
After setting the camera I should * align the camera to the scene *. If you are familiar with maths of transform, you will notice that it is identical to * align the scene to the camera *
It will be good if I can rotate the ground grid. How I cannot. So I created a virtual cube called "coordinate"
Then I selected the camera gameobject, Menu-GameObject-Align View to selected. Now we can edit the scene from the perspective of camera.
Now ,select the "coordinate", move, rotate and scale it until it fits the scene.
- One cannot perfectly fit because the error in fov and the lens distorsion
- donnot worry the z-position (distance to camera) of the coordinate. You can only align the xy-position, since the increase in z-positioin is identical to the decrease of size.
- according to maths, you can synthesis z-rotation by x-rotation and y-rotation. There is only 2 degree of freedom in rotation.
Finally, I will show you the magic.
Since the "coordinate" is aligned with the camera. You can build the ground or something else under the transform of coordinate. Now we want the coordinate transform to be the world transform. How to do that?
- Drag the camera into the coordinate gameobject
- Reset the coordinate's transform to 000 position, 000 rotation and 111 scale
- Drag the camera out of the coordinate gameobject
The unity engine will automatically calculate the camera transform if the camera was transformed by the same transform that transforms the coordinate to world.
Finally we will decide the actual size of the scene
- Drag the character into the scene and make sure her transform is identical, and she is not under coordinate gameobject
- Drag the camera into the coordinate gameobject again
- Scale the gameobject until you are satisfied with the character's size
- Drag the camera out of the coordinate gameobject.
Congratulations! You have finished setting up the Camera.
Fitting in the obstacles
As I mentioned before, From 2D picture one cannot get enough information about the object's actual position. One cannot tell if it is a small object near the camera or a big object far from the camera. Luckily we can infer more relationship form the shades and common knowledge.
The common knowledge I used is the fact that most of the objects are on the ground. They are not floating in the air. With this addition information I can obtain the object's actual size and distance.
- create a cube
- assign it with Cutout material
- Wrapped it with another gameobject called "CubeObstacle". Make sure the cube's local transform is (0,0.5,0;0,0,0;1,1,1). That makes the gameobject's origin is on the bottom of the cube
- Make the "CubeObstacle" a prefab
- Also, you can make more prefabs such as cylinders or horizontal cylinders or spheres and so on
- Create a plane with Cutout material and name it "Ground"
- Make sure the plane's position is 000. Then size it bigger.
- Select the camera, Menu-GameObject-Align View to selected.
- Now From the camera perspective, drag the obstacle prefab from the project tab onto the Ground gameobject. That will make sure the obstacles are on the ground. So you can drag them to the correct position according to photo
- Size and rotate the Obstacle to fit the photo. Donnot size the cube gameobject, size the CubeObstacle gameobject.
Additional Mention
- Donnot forget to set up the lighting correcting. I used a directional light for shadows. And I use photos to make a cubic skybox for ambient. You can use the same photo for 6 sides of the skybox since skybox mainly provides low-frequency lighting informations.
- Donnot use your dirty uncleaned dormitory to take photos. Art is someting atrificial instead of natural. Actually I cleaned my dormitory first then mess it up manually, to carefully design a game level.
Time budget three hours. Traffic used additional four hours. Use fl studio mobile make bgm .
Wake up and you haven't start to make the music yet :n
I think 15 is a good number for player hp, isn't it?
