Augmented Images is a brandnew feature (as of May 2018) of the ARCore framework. Although something like this exist for quite a long time with other frameworks, it is very new for ARCore. In the following slides we jump into how to use Augmented Images in our own AR App.
We start with how to select what images should be tracked by ARCore.
This is defined by the AugmentedImagesDatabase class. To let ARCore track images in that database, the followin steps are required:
Config instance (this is a session configuration)onUpdateListener with the scene of the ArSceneViewthat is called on every frameAugmentedImageNode (your own class implementation that extends AnchorNode and renders your 3D model) with the 3D model to show for every image that was foundAugmentedImageNode, so it keeps track of the position of the image in space and renders the 3D model in placeIn the following we will go step by step through the code that is required here
The following example shows how to setup the AugmentedImageDatabase and add an image as Bitmap:
private boolean setupAugmentedImageDb(Session session) {
AugmentedImageDatabase augmentedImageDatabase;
Bitmap augmentedImageBitmap = loadAugmentedImage();
if (augmentedImageBitmap == null) {
return false;
}
augmentedImageDatabase = new AugmentedImageDatabase(session);
augmentedImageDatabase.addImage("delorean", augmentedImageBitmap);
config.setAugmentedImageDatabase(augmentedImageDatabase);
return true;
}
Here you can see how to load an image as Bitmap
private Bitmap loadAugmentedImage() {
try (InputStream is = getAssets().open("delorean.jpg")) {
return BitmapFactory.decodeStream(is);
} catch (IOException e) {
Log.e(TAG, "IO exception loading augmented image bitmap.", e);
}
return null;
}
Next a Config instance is needed to configure the AR session with the AugmentedImageDatabase:
private void configureSession(Session session) {
config = new Config(session);
if (!setupAugmentedImageDb(session)) {
Log.e(TAG, "Error when setting up Image Database");
}
config.setUpdateMode(Config.UpdateMode.LATEST_CAMERA_IMAGE);
session.configure(config);
}
Next an example for a OnUpdateFrameListener, that adds maps the images to the 3D model. The configuration of the session is only done once as soon as a session is available, not every frame.
private void onUpdateFrame(FrameTime frameTime) {
ArSceneView arSceneView = fragment.getArSceneView();
Session session = arSceneView.getSession();
if (session == null ){
Log.e(TAG, "Session is null");
return;
}
else if (config == null){
configureSession(session);
}
Frame frame = arSceneView.getArFrame();
Collection<AugmentedImage> updatedAugmentedImages =
frame.getUpdatedTrackables(AugmentedImage.class);
Log.e("onUpdateFrage", "Updateing Frame");
for (AugmentedImage augmentedImage : updatedAugmentedImages) {
if (augmentedImage.getTrackingState() == TrackingState.TRACKING) {
// Check camera image matches our reference image
if (augmentedImage.getName().equals("delorean")) {
AugmentedImageNode node = new AugmentedImageNode(this, "andy.sfb");
node.setImage(augmentedImage);
arSceneView.getScene().addChild(node);
}
}
}
}
Next the OnUpdateFrameListener can be registered in the onCreate() method like this:
fragment.getArSceneView().getScene().setOnUpdateListener((this::onUpdateFrame));
The rendering of the 3D model onto the image is done by our own class called AugmentedImageNode. Here it is:
public class AugmentedImageNode extends AnchorNode {
private static final String TAG = "AugmentedImageNode";
private AugmentedImage image;
private static CompletableFuture<ModelRenderable> modelFuture;
public AugmentedImageNode(Context context, String filename) {
// Upon construction, start loading the modelFuture
if (modelFuture == null) {
modelFuture = ModelRenderable.builder().setRegistryId("modelFuture")
.setSource(context, Uri.parse(filename))
.build();
}
}
/**
* Called when the AugmentedImage is detected and should be rendered. A Sceneform node tree is
* created based on an Anchor created from the image.
*
* @param image captured by your camera
*/
public void setImage(AugmentedImage image) {
this.image = image;
if (!modelFuture.isDone()) {
CompletableFuture.allOf(modelFuture).thenAccept((Void aVoid) -> {
setImage(image);
}).exceptionally(throwable -> {
Log.e(TAG, "Exception loading", throwable);
return null;
});
}
setAnchor(image.createAnchor(image.getCenterPose()));
Node node = new Node();
Pose pose = Pose.makeTranslation(0.0f, 0.0f, 0.0f);
node.setParent(this);
node.setLocalPosition(new Vector3(pose.tx(), pose.ty(), pose.tz()));
node.setLocalRotation(new Quaternion(pose.qx(), pose.qy(), pose.qz(), pose.qw()));
node.setRenderable(modelFuture.getNow(null));
}
public AugmentedImage getImage() {
return image;
}
}
The Augmented Images allow to track images, add anchors to them and render stuff on top.
This allows you to easily augment your world around, just take pictures and add them to the image database. But it does not work with all images. If you want to know if an image is easy to track, ARCore provides a tool called arcoreimage tool.
A good reference image is hard to spot with the human eye.
Run arcoreimg eval-img to get a quality score between 0 and 100 for each image.
Google recommends using images with a score of at least 75.
To test an image quality run this on the command line:
./arcoreimg eval-img --input_image_path=dog.jpg
The arcoreimg tool also supports creating image database from a directory or from a list of image files. This might be handy in order to easily add a buch of images to your app.
There are plenty of things to try out with Augmented Images. It’s a lot of fun!
Find here more information on the Augmented Images Feature