THE ASSET PIPELINE EDITOR - PART 3

On part 2 of the series, I talked about some basic features of the APE, so it's time to show how to change the structure of a project and populate it with some raw files, that is, before starting to build assets.

I could begin this post by telling you how to create a new solution from scratch, but instead, I'll show you a nice small feature the editor has that allows you to open recent solutions, which spares you from having to use the Open Solution menu option ("Control + O") and browse folders until you locate the solution file.

Do you notice the small red (almost transparent) cross beside the only entry in the list of recent solutions? If you press it, you will be prompted to confirm entry's removal from the list. The cool thing is that APE does not need to restart in order to refresh the list. Ditto for "Clear All ..." option.

Once the solution is loaded we can start modifying the tree structure to meet the needs for our game. This is how the solution looks like so far:

Let's add a couple of nested folders to the Windows_Own container by selecting the "Add Folder" menu option (or clicking "Control + Shift + F") twice. Now our solution looks like this:

By default, folders will be created with the name "folder_<#>", so let's renamed them with a meaningful word (by double-clicking the node):

Better, right? But wait a minute! What if we want to have a similar structure on the default container? All you have to do is open a context menu for the node you want to copy (right-click over "textures") and you will be presented with this:

\

You have two options there: either you move or copy the node and all of its content to another non-nested location; in this case, to the Windows_Default container. Note: if you try to move or copy the "images" folder into the "backgrounds" folder the operation will fail.

Since we want to replicate the structure, we then select the copy option. Then, by opening the context menu fro the default container we will get the following:

By pressing paste, the new structure for our solution will look like this:

It's important to mention here than we could have obtained the same result by using drag'n'drop. Yes! Dragging the textures folder and dropping it onto the default container would have been allowed. In fact, when you drag'n'drop nodes in the solution explorer, the APE asks us what we want to do with them: move, copy or cancel.

Also, it's worth mentioning that, in this case, we could have copied the whole content of the self container ("Windows_Own") by using the context menu for the container. That is also allowed. The difference is that in this case the container it-self won't be moved or copied, just its child nodes.

Now, let's add some raw files. Shall we?

There are three ways to add raw files to containers or folders: (a) you select the container/folder and then use the "Add Raw File" menu option/button (or press "Control + Shift + R"), (b) you select the container/folder, open the context menu for it and click on the  "Add Raw File" option, or (c) you use drag'n'drop (you can import raw files into your solution by dragging them from the explorer and dropping them onto the container/folder).

In the picture below, you can see how to use the context menu for the target folder.

One important feature to mention here is that you can import more than one file at a time (batching) with any of the three ways the editor offers, including drag'n'drop.

For the example, we just added a nice cloud background to the backgrounds folder:

The picture above shows a lot of new information about the editor:

1. The name given to the added raw file is its filename (without the extension, if any). This is true provided that the name does not exist in the same "relative" location of the solution's structure, or else it will be renamed to "file_<#>".

2. On the container tab, the editor will show you the proper previewer for the file (remember that if none is found it will just show the system icon for it).

3. A new tab gets enabled: "Raw File". Here is were you tweak import settings for the raw file. The APE will try to assign an importer to it based on the file extension, giving priority to the importers marked as default (more on this on later posts).

4. The source file is copied into the "Sources" folder for the solution. This folder works as a repository for the entire solution, so you cannot have two source files co-existing with the same name on the folder since one of them will be replaced. This has nothing to do with the names you give to the nodes on the solution explorer (you can have two nodes with the same name in different "relative" locations on the solution tree).

In order to change the name of the raw file we could do it in the same way as before by double clicking on the node, but in this case we will use the "Name" field on the raw file tab and rename the node to "bkgClouds".

Now, before showing you how the structure of the solution currently looks like, let's add a new project for, say, the iPhone platform. And here it is ...

As you can see the default containers on both projects have the same exact structure; however, the self containers are different. In effect, that is the purpose of self containers.

Although the "bkgClouds" raw file is present in both default containers, it is treated as a deep copy; so mirror operations are bound only to add/remove/copy/move. For properties, like importer, processor and formatter-like settings, each node is independent, so you can have one set of settings for the raw file for the iPhone and a different set its sibbling on Windows. Really great!

Now, what if you want to import two different source files but keeping them under the same raw file on different projects? Let me translate it with an example ... say that you want to replace the source image for bkgClouds but only on the iPhone project with a lower_res version, well ... yes, you can! How? Simple. Look at the picture below:

On the Raw File tab there is field named "File" with a button labeled "..." (marked all in red), which shows the location to the source file in the solution. When you click on the button you can browse your folders until you get the file you desire.

In this case, we'll select the 320x200 texels version of the cloud image for the iPhone platform to get this window:

The APE will prompt us to confirm the operation and also it'll let us delete the existing source file if needed. Since this is not the case, given that we need both of them (one for Windows and the new one for iPhone) we let this checkbox unmarked.

Once the operation completes, the image previewer for the "bkgClouds" raw file on the iPhone project will show the new information. Now we have two different source files for the same logical name. And therefore, after building assets, when you load the "bkgClouds" texture on your game, you will get the 640x400 version on Windows and the 320x200 on the iPhone. No need to add "_640x400" or "_320x200" to the load operation.

Let's have a look at our solution's Source folder:

As you can see, the respository has both source files for the same raw file. And I guess that you now fully understand the difference between "source" and "raw"" file (if not, the former is the actual file and the latter its representation on the solution's tree).

Ok, before finishing this post, let me show you one more relevant feature the APE offers. For this task, say that you accidentally messed with the Source folder and deleted one of the images; for instance, the one ending in _640x400. What would happen? Well, when you re-open the editor, you will see the following:

The APE is indicating you that the solution is corrupted and also marks the path to the offending raw file. To fix it, you can either manually copy+paste the source file to the Sources folder or you can use the Raw File tab -as we did before- by clicking the "..." button beside the "File" field and browsing to the location of the source file.

In order to valdiate the solution you have to options, either you save, close and re-open the solution (in which case the APE will warn you that you are attempting to save a corrupt solution) or use the validate menu option ("Control + Shift + V") ...

Having finished the validation processed, the editor will look like this:

So, we can now safely continue to work normally with our solution and save it whenever we want.

Btw, you can browse to the folder where you solution is located by using the "Open In Explorer" menu item (ditto for projects) ...

And you'll get to ...

Do you notice the icon on "MySolution"? When the editor is closed you can open it and also load the solution by clicking its file. Indeed. The APE will open and load the solution for you.

Ok guys, this is all for this post. On my next post we'll talk about building assets.

See ya!
~Pete

THE ASSET PIPELINE EDITOR - PART 2

Continuing with the second part of the series, and before writing about technical stuff (the coding side), let me introduce you to the editor it-self.

When the Ape starts, the editor looks like the picture below:

By default, the only tab that is enabled is the one named "General", where you can configure general settings to control the editor's behavior. There are a few options that speak for them-selves, like "Automatic check for updates", so I will refer to others which I believe deserve a few words since they will affect each solution/project created in a cascade manner:

1. Default Main Folder: every time you create a new solution, this is the folder that will be shown by default on the Folder dialog. Think of it as the directory that will hold your solutions and projects.

2. Default Root Folder: for every project created, this is the relative root folder assigned by default to each container. For those like me who come from XNA-based games, this is usally set to "content".

3. Default Build Action: this is the action to execute when building assets from raw files in solutions/projects, which can be "Build Always", "Build If Newer" and "Do Not Build".

4. Default Copy Action: this is the action to execute when copying assets after a build action finishes on a project, which can be "Copy Always", "Copy If Newer" and "Do Not Copy". You will see later that this also refers to "moving assets".

5. Available Platforms: this is the platforms that a solution can support and you can add/remove custom platforms to the ones provided by default. The one marked with an asterisk "(*)" indicates the project that will be automatically added when a new solution is created.

6. Default Writers Per Profile: this control has two purposes; you can add/remove/modify custom compilation profiles as well as the write units assigned by default to each and every one of them; again, the compilation profile marked with an asterisk "(*)" indicates the one that will be selected by default when a new solution is created.

When you add a new profile, you can set its name, the relative folder where assets will copy/move when built, and the default write unit for it. We'll get to the concept of a write unit on a later post.

7.  Default Importers Per Category: every import unit corresponds to a category (that you can set when creating the importer), so what you can define here is which will be the default importer for each category. When you add a raw file, the editor will find the category it fits in and from there it will try to assign the default importer; if no importer is found, it will move to a default category and importer (pass through).

Note: both, import and write units are plugged into to the APE as add-ons, so when you execute the tool, the editor will automatically create the collection of available import and write units.

8. Save/Reload Settings: when you change general settings you have to options: either you save the settings for future use or you reoad settings previously saved; if no settings are found (or saved settings get corrputed) the APE will generate and save default settings for you.

Now, what are Solutions and Projects?

A solution is the root node in the APE's logical tree which will directly hold projects as its child nodes. In other words, is the root node that will contain asset projects.

A project is the node that will contain the structure that will be used to build assets for a specific target platform. You can have as many projects you want on a solution but only one per target platform.

When you build a solution, all the project it contains will be built (unless you specify not to build one or more projects). Conversely, when you execute the build action on a specific project, the other will remain intact.

In order to create a new solution, you can open the "File" menu and select the "New Solution" option, click on the "New Solution" icon on the main tool bar, or press "Control + Shift + N".

When you do that, a pop-ip window will prompt you to enter specific information like the name to give to your new solution, the folder where the solution file will be saved, whether a subfolder with the name you enter must be created or not and the "Output folder".

The latter is quite important since it indicates the path to the root folder where all assets generated for the default project will be copied or moved. It works like this, say you have a Visual Studio solution for a game you are developing (or a repository for the assets, not the raw files- to be used); you browse to this path, you select it, and from there the APE will append at build time the relative folders corresponding to the selected compilation profile as well as to each container, when it corresponds.

The initial structure of a solution is the following:

Being the only project created the one that corresponds to the target platform inidicated as the default one on the general settings.

As you can see in the picture above there are two more nodes that are created by default within the project: the initial containers ending in "_Default" and "_Own".

The former holds raw files shared with all projects. Thus, when you add a new raw file to that container, a new node will be also added to the default containers in all remaining projects, with the same name.

And the latter holds raw files assigned only to that project. So, this container is where you put raw files that you don't want to share with other projects.

Now, there is a third type of container: partial containers (to which I also refer sometimes as "shared" containers).

Say that you have three projects in the solution (Windows, XBoxOne, PS4) and you want to share content on the first two (Windows and XBoxOne); well, in order to do that you create a partial container for the windows project, add the XBox One platform and presto!

To add more projects to your solution you either select the "Add Project" item on the "Solution" menu, you click the "Create Project" button or press "Control + Shift + P".

Again, you will be prompted to specify the output folder for the new project.

Ok, moving on ...

When you create or load a new solution, you will see that other tabs will get enabled: "Solution", "Project" and "Container".

Many options in the Solution tab are similar to the ones found in the General tab; in fact, this is where you start to benefit from the cascade approach given that you have the possibility to override the default values given to each "shared" option.

But there is one more relevant option you can set for a solution: "Move Assets". The only explanation I will post for now is that when you build assets, the APE save them on a specific repository for each project and then, copy or moves each asset to its final destination folder (the one located in the output folder for a project).

For projects, the logic is the same as before, tweak parameters to override default values coming from the solution or general settings.

Here you can also indicate:

1. Group Id: even though you can set different behaviors per platform, you can also indicate affinity among specific platforms in order to set the course of action for import/write units. For instance, for XNA-based projects, and id of zero ("0") would mean a "HiDef" profile meanwhile one ("1") or above would mean "Reach" profile.

2. Copy To: remember that for new solutions you had to sepcify the output folder for the initial project? Well, you will have to do the same for the new project. So, each project can have its own output path, which is strongly recommended so that you avoid the risk of overwriting asset files.

3. Default Writers: not only you can override the values that come from the solution or general settings, but also you can change the folder that will be added to the output path when building assets for a specific compilation profile (like Debug, Release, etc.).

Regarding the Container tab, there is no much you can do there other than change the root folder, which is initially set to its default value (usually, "Content").

For default and self containers, names cannot be changed. For partial containers, you can set a different name, and you will also get a list of the projects it shares content with.

There is another tab that is also activated when a solution is created or loaded ("Writers") but I will refer to it on my next post.

Before reaching the end of this part, I would like to refer to the search tool, which you can use when you want to find nodes in the solution that start with a specific character or group of characters ("Control+F").

The beauty of this control is that you can do a new search recursively, that is, on top a previous search. So if you first search for "Windows" and then for "myImage", you will get all the nodes (and its children) that start with "myImage" within the nodes that start with "Windows". By the way, the top-most node (the solution) is not considered for this feature, so the result it would have been the same if you had entered "my" instead of "myImage".

By pressing the "Reset Search" button or "Control+Alt+F" the whole solution is again displayed on the explorer.

This is a powerful tool and I'm planing to extend it with additional features in the middle run.

Well, that's it for now. On part 3 I will talk about adding raw files, compiling projects and a few more useful features available on the APE.

Stay tuned,
~Pete

THE ASSET PIPELINE EDITOR - PART 1

Now that the cat is out of the bag, I believe its time to start posting some details about the Asset Pipeline Editor (from now on, "the APE").

So, what is this tool? To answer it, let's travel through time to a point, say, four years ago or so.

To make the long story short, during the golden age of XNA I was in the search of an efficient way to replace the built-in content manager class with my own. So I created my own version of this class, but I was still using XNA's content pipeline.

Then, when MSFT announced that they would stop any development on the XNA Framework, like many of you, I remained captive of Visual Studio 2010 IDE for building xnb's. During this year, I decided not to wait any longer for miracles and start building my own content-pipeline replacement.

Let's face it! How many of you have been lately crying out loud for XNA 5? Or asking where the content pipeline has gone? You still need VS2010 to use the content pipeline if you want to use XNA, Monogame or ANX. And no definitive solution has been provided yet. And even if there is something in the works, is it worth for teams?

Be honest here. You guys know that the in game development a content pipeline is indeed needed, but why sticking to a particular IDE for programmers (like VS) or any other programming IDE in the first place? It could make sense for programmers only, but for teams, artists, or even for solo devs that want to have an independent process, it does not. In fact, it could turn out to be cumbersome.

So, that is why I developed the APE. To replace XNA's content pipeline "in spirit" (since it has key deifferences) for those inhouse projects where I wasn't using Unity, UDK or any other authorware with its own content management features.

I must admit that at first I wasn't expecting to get these results, since I was aiming lower, but as times went by, I realize that -and please allow me to say it- I was creating something really good. So I went on until I said "Wow!".

Now, about its key features:

1. Its not for XNA, only: it works for any kind of custom content, not only xnb's. In fact, if you have your own way to manage content when programming a game or provide solutions (like WaveEngine, for instance), you can use it safely since the tool ends its job when the asset its build and copy to the folder you indicate.

2. It's a highly customizable tool: not only you can tweak the editor to meet the needs for your project (to some extent, of course) but also you can define your own building process: import, process, format, writing. You dream of it. You got it.

3. It eases the task of managing game content: that's it, throughout the whole development process. If say, you just want to use the input files as is, like pngs, jpegs, and so on so forth, without any processing, you can because the editor comes with pass-through units. So you can still use the tool to manage which content goes where, even if there is no processing required.

4. It's independent from any programming IDE: this is heaven for artists! If you are a solo programmer working on a game, having to use VS to build content is fine, but when you're woring on a team with artists this is not good at all. The APE comes with a GUI of its own.

5. Test before you promote units: "why is my custom processor not working?". Say bye-bye to these kind of questions since you can test your custom "units" (as I call them) before using them with the editor. Indeed. You can build your own testing assembly for your custom content/process. And once you give it a green light you plug it to the editor as an add-on.

Here you can see the main editor as of today:

The areas indicated in the picture above are the following:

1. Solution Tree: create a solution, add a project for a platform, and you will be able to traverse all the nodes here. Projects can have two or more containers: self, default and partial. And you can add as many folders as you want to each container. In future parts I post more details about this.

2. Search Tool: if you need to see specific nodes of the tree this is a quick way to do it. It works recuresively over the last search, so you can go on reducing results until you find what you want.

3. Output Settings: for every "raw file" that you want to import, you can define how to process it and format it. And for each project, you can define how to write ("export") assets to disk. These are powerful tabs. More about them later ...

4. Build Tool: here is where you decide whether to build assets for the whole solution (all projects) or for a specific project; and also where you define the compilation profile: debug, release, test, ... you name it ... add the ones you need for the solution. You will be able to watch it on a how-to video, later.

5. Configuration Tabs: so far there are four: general settings, solution settings, project settings and container settings. Basically, you can tweak many properties there and even add/remove entries for platforms, profiles, default importers, default writers, to mention just a few. In a later post I will cover all settings.

6. Log Panel: classic in any respectable IDE; here is where the editor informs you about the state of a given process, whether it succeeds or fails, warnings, exceptions. The usual stuff. It's a real friend to understand what's going on. There is more to it than meets the eye, and that deserves another post.

7. Bars and shortcuts: well, this is not indicated in the picture above, but of course that you have menu items (both, as text and buttons) and key shortcuts for many features. Not to mention, contex menues, where applicable, for example to copy or move content. Ahhh, yes! I almost forget it, drag and drop is allowed.

I don't want to go beyond the scope of this post, but I cannot help adding the following picture:

Above there's an example of a test assembly. When you create your units for a specific type of asset- by the way, using C#, this is where you debug them and see whether everything works as expected. Add break points, switch over text, find offending code!

And finally, for this post: "previewers". When you traverse the solution tree and select a raw file (or source file), the editor shows information about the file as well as a previewer for it. By default, the APE comes with previewers for some image, audio and video formats. Or else it will switch to an icon previewer. But the good news is that if you need to preview more formats you can create your own previewers! What is more, you can replace all built-in ones if you want ....

The picture above is an example of a previewer for an audio file.

Now, before ending this post there two remaining points I would like to address:

1. Current State of the APE: ready for my inhouse projects: the units, besides the pass-through ones, are meant for my inhouse projects, only. Plus, there are some features I would like to add (and even some porting to do) before a public release, if you guys are interested in having this tool, so ...

2. What's next: my idea is to start a fund-raising campaign on IndieGoGo to make a first release of the tool,  and some extended goals like developing some units for XNA'ers and Monogamers (so they don't have to), and even doing some porting to other platforms like MacOsX and Linux (since so far the tool only works on Windows for .NET 4.5). And yes, the link at the end of the trailer is not working right now as the campaign is currently on draft mode.

So, this is all for now.

I hope this post sheds some light on what's the APE about and that you guys are interested. It's upto you guys to define whether this baby eventually sees the public light (if not, I'll continue to use it for my inhouse projects). Your call ...

'till next posts!
~Pete