Outerra Worlds Forums
Middle-Earth World => General Discussion => Topic started by: bburns on October 17, 2019, 04:39:39 am
-
Hello all, I've started playing Adventures in Middle Earth and was interested in making some landscapes to make exploring the world more interesting - then I wondered if someone else had made a DEM for Middle Earth, and found the incredible work that you guys (monks) did. So I got the 10k DEM file [1] and the Shapefiles [2] and put them into QGIS and with some coloring and hillshading have this so far -
https://imgur.com/M8f83o3
What took the most time was aligning the DEM and the vectors - the georeferencing plugin didn't seem to work very well, so I tried editing a world file for the DEM directly and importing it - that worked a bit better, though took many tries to get everything lined up, and it's still not perfect. The current world file has -
200.1
0
0
-200.1
-900
2001100
Some things to do -
- add linear mountain ranges for labelling (just did a few so far)
- add regions for labelling
- try some different color and labelling schemes
I'll post updates here if I make more/better renders - I'm still experimenting with colors and styles.
Thanks for all the great work you guys have done - I hope this project keeps going!
[1] http://worlds.outercraft.com/forum/index.php/topic,27.60.htm
[2] https://github.com/jvangeld/ME-GIS
Brian
https://brianburns.org
-
Hi, cheers for that. Looks great!
Maybe I should release the data. I no longer work on it or have to time to work on it. I do still have it--- I'd have to dig it all out. It would require that someone using the data have Global Mapper- the project is in a very old version of Global Mapper...and I'd prob have to point it to all of the files again as it has all been split up and moved from the original directories. Wouldn't be straight forward task but I certainly have all of the shp files and the dems. I couldn't do it just at the moment though either as I'm busy in the middle of something else. I wouldn't want to get sucked back into being 'support' for a project either...
monks
-
Hi Brian, I think Monks had the final data but I think I still have the files we used to create the data but it wont be any good to you. Between us I think we could recreate the data set that we got to the final state.
Personally I think it might be a good idea to release the data in whatever form we had it in as were both no longer working on it now. But I agree with Monks in that to use it outside of the weird system we had you really need to use Global Mapper.
In a sense we released the data for Outerra because it was a proper 3D realtime engine that made it possible for people to see it without having to faff about with the data set much to get it into a form that someone could use. Outerra was an app that fitted perfectly in with our data as at the time we had no real time system to display it properly.
Red,
P.S. Hey Monks, long time no speaky ! Will have to catch up :)
-
I sent you a text the other day Robes...to your moby...prob no longer your number though. Yeh, sent you an email ages ago but your emails have always been iffy so never knew whether you received it or not... ;D
I think the aim should be to get it into Outerra again...clearly...but with a single project in Global Mapper. So it demands that someone in the project has Global Mapper. So they'll have to put their money where their mouth is if they want it to happen... ::) ;D
monks
-
Hi monks and Redrobes - I didn't realize you had written - I thought I was subscribed to this topic!
My main purpose was to make some detailed maps to play AIME with - like down to the 1 meter level. That would involve procedural generation, which I'm not very familiar with. But I figured I could feed it your 10k DEM (200m/px) as a starting point and have it fill in the blanks.
Then I got interested in making a nicely labelled map, and played around a bit with the shapefiles - adding region and mountain layers with curved labels and fontsize in km. I put that all on Github here https://github.com/bburns/arda, minus the 10k DEM - it's still a little preliminary.
Anyway, I'll be switching now to the procedural generation, but could post a map to reddit r/tolkienfans to see if anyone was interested in helping maintain the project. And I haven't written to jvangeld yet - will do that also.
Thanks again for the awesome maps! :)
-
That is a really nice render. Between you and Hellricer the GIS files are really improving.
Just to get this ME-DEM reunion really cooking, I sent SeerBlue an e-mail.
-
:D I'm still in contact with Seer on FB. I have a project that works with Global Mapper 14- which is the GM I have. I have World Machine projects in WM 2.3 Professional 64 bit edition.
Global Mapper contacted me yaknow two years ago because they were looking for projects that they could showcase using their software. Nothing came of it- probably because the project was in remission.
To make it happen it needs the following:
Someone other than me with Global Mapper.
Someone other than me with World Machine.
Someone other than me with Outerra.
HFZ support being reinstated in Global Mapper- speak to Mike over there. If someone gets a recent copy the support for it almost certainly won't be in there anymore. I did get Mike to reinstate it once after he inadvertently forgot to include it- there are like over a 100 formats that it supports.
It is possible to do it without HFZ support but HFZ is way better than the other option .bt.
Someone with a powerful PC. The World Machine renders that knit all of the output from GM into a tile set for Outerra take days. World Machine adds all the erosion and blends stuff together.
Obviously it would need Brano's support (mapressor) getting it into Outerra. I *think* he would be ok with it given enough interest. Someone also needs to run mapressor- but most people with Outerra could do it.
I do have the workflow I used in a text file- not that you'd need to follow it exactly...you might get away with not using World Machine...but I can't see any other program out there being the right fit for the job- all round.
The project would need storage- we used to use Oshyan Greene's ftp...but something else would be necessary now.
That's just to touch base camp on it.
I think 2013 was the last year we worked on it...possibly into early 2014..I've got the project spread over 6 hard disks...backups...it appears I've probably got it duplicated 2 or 3 times. The last Global Mapper file is dated early 2013, a zipped format .gmp at 3.7 GB.
Me, personally, I think Outerra is the community to pitch this at. It's the only renderer that could do it justice- sheer scale, and geoerffed.
monks
-
I'm going to just cut to the chase and post about this on the Outerra Boards. I'm pretty sure I've got all the data and I know what's needed. I'll probably drop Brano an email too since I think the Outerra community is the best place for this.
monks
-
I hope you find some maintainers - I posted a map to reddit, and mentioned that you're looking for people - https://www.reddit.com/r/lotr/comments/dncyxc/map_of_middle_earth/.
And we moved the curved labels to jvangeld's ME-GIS repo, and I added the 10k DEM to the arda repo, so other people could play around with it - https://github.com/bburns/arda.
I might not have much time for the map for a while, but hope to get back to it eventually!
-
Yeh, don't panic Brian, this release of the data was not aimed at you! ;D lol You've done some good stuff with the maps. Any interest is great. I've posted a link to your stuff on FB.
Personally I doubt if anyone will be forthcoming, but it's worth a go. Brano has offered to put the data on their server. He also mentioned a Google Drive. So, I think hosting of the data can be worked out.
monks
-
Me and Brano will split the project storage over 2 Google Drives around 65 GBs of space.
monks
-
Me and Brano will split the project storage over 2 Google Drives around 65 GBs of space.
monks
That is really great news!
Forgive me for being pesky, but I think this nicely illustrates why people should always thrive to keep data of this character in open formats...
Keeping the works in proprietary formats makes you "vendor locked-in" to the software (in this case not exactly cheap one either), which in this case even obsoletes some crucial features in latest versions. I completely realize that often times there might not be good-enough alternative — I'm just making a general remark. Shapefiles, .bt files, etc. are IMHO step in the right direction.
Anyway, I did manage to get a copy of GM v13 and I could open some example HFZ files from L3DT website just fine.
That being said, I know absolutely nothing about how to use the software yet, or what is actually unfinished or "in-progress" :D But I would be very interested in learning as much of the workflow as possible!
I can't get Outerra running, but that's something I can live without for now.
-
ok cool...that's one half of what's needed. Storage is there now. Uploading it all will take weeks though.
Mike at Global Mapper -reinstate hfz- sure he'll do it....ahh yeh hfz fine. cool.
The other half is World Machine.
The whole idea was to use industry standards and treat the map as if it were 'real'- that is use GIS mapping software. Because, as Tolkien stated, M-E IS Earth but set back in history- an alternate timeline. There was no proprietary format of course for it.
monks
-
I'm uploading the data to my Google Drive. It will take another two weeks at least. There will be a fair amount of data in there that is not necessary to have. I need to organize it once it's on there. I also intend to reconstruct the Global Mapper project file and the World Machine project file. All of the files will be missing from the directories so I'll have to reconstruct a folder with the files in them and point the apps at them. So. we're talking prob 3 weeks on this at least.
monks
-
Respect and appreciation to you, Monks.
-
I've just opened the global mapper package file in GM 14.2.3 which was the latest one I used in the workflow. It is a zipped .gmp file at 3.6 GB. Not tested its compatibility with the latest version of GM. All of the necessary filers are zipped in that file. There are a vast number of other files and packages and projects which are not included in that file as they are not needed for the Outerra build run. As a whole all of this comprises the project data.
There is also a workflow .txt which details the workflow I used. There are also lots of other procedures and processes outlined in that too.
I'm going to have poke around today and determine which files are the central ones to the project. Get some kind of idea of what's what.
-
Hey monks, is there any update? I'm mainly interested in the workflow textfile you mentioned to get better understanding of the overall process.
I had set up both GM and WM and have made some progress in learning how to use them.
Also, I'm still a bit confused about the 10K DEM - I noticed that it's combined from four parts with quite noticeable "seams" in the middle. As I understood it, it was generated from vector contours data, right? Are those available anywhere? (I suppose it's not the 'Contours_18' layer in GIS, which is much less detailed.)
Thanks a lot!
-
The stuff is on my google drive. I need to go back in and find the latest project again. I did find it and I marked the hard drive with "latest" :D So, I could give you access but it would prob be confusing and frustrating for you right now. It wouldn't take me long to track the project down and identify it. There's other stuff too which ism;'t on the google drive yet but it's more periphery- not core. I need to sit down and identify which projects are the latest- it ended up being split into multiple projects. There is one huge one which won't open -but I splintered projects off from it. So, I can do it at the w/e or before and let you have access to the drive and any updates you if I have any.
monks
-
You got a gmail address mate?
monks
-
sent you pm
-
ok you got access...I'm looking at the hard drive again with the latest dataset on it...will let you know when I pinpoint the most relevant files.
monks
-
Latest files looked at by me...
workspaces (gmw) and package files (gmp)
2011_Update149.gmw
2011_CurrentWorkspace.gmw
New_Mountain_Extents.gmw
New_Mountains_Extents22.gmp
One of these may be the very large one which had everything in it- which didn't load. The other ones are subsets of the larger one.
All of the shp files are stored in them but also separately too. There may be shp files in the external folder that are not in any of the gmps or gms..quite possible.
I suggest downloading the entire folder. There's a file missing from the first workspace...not sure if that's important...
monks
-
I've found the missing file from the project. I'm in the process of uploading another set of folders today. They may duplicate some stuff in there but I do know that there is some stuff in them that is not in the existing folders.
monks
-
Awesome, you're a saint. I opened the largest gmp file without a problem and it seems that most of the layers are there.
I'll now try to get familiar with the data. The amount of work behind this project is just mind-bending...
-
Cool...which one are you referring to? What's the name of it?
monks
-
I tried New_Mountain_Extents.gmw and New_Mountain_Extents_20.gmp. The latter seems to be most complete and loads without any errors.
-
Ah yes, tracking stuff down at the moment. I think there may well be other files that are not on the google drive right now. I will take a look at the listed files here.
The Minecraft team are considering moving over to Outerra btw. They contacted Brano. Did I mention that they wanted to use the project? If it goes through it could be good news for MEDEM. They're going to warp the data in Photoshop using a smart object. What that means is we can warp the layers to using the same warp settings in the layer, after the fact- just drop things in and they will be auto warped to the exact same settings. It will all be done at 20K. Some info will be lost from downsampling the 40K layer but it's worth it. That layer is just used as 'fill' between the other layers.
The non-terrain layers (vector layers) can be warped as rasters and then converted back into vectors (.shp files) painlessly using software like WinTopo.
I'm looking at restoring my WinTopo license at the moment. I did find a license key they sent me on my backup hds. Last time I used it was 2012!
monks
-
I just opened the file. There are absolutely loads more layers than that hellricer. For starters there's no terrain in there. There are prob 150 vector layers missing from this and all the terrain which includes the base dem, the mountains, Mordor and the fill layer. They were all kept separate in the end because of the sheer size of it. So it's a case of tracking down the terrain layers that are the most recent. All of the .shps layers are in the SHP folder...they're the vectors.
monks
-
Copy of message in the GD share:
Hi Robert, going through files to send to the Minecraft team who want to use the data. This is the foundation topo. This file comes with the tfw and projection info so be sure to also download those too. You already have access to this directory but I'm just giving you the headsup as I go through these over the next few days. Those project files mentioned on the Outerra boards are a bit crap. There are way better ones.
Map88_20k. There are also .prj and .tfw as mentioned.
monks
-
Hey, monks, thanks for the heads up. I'll definitely check it out.
I can give some update although I didn't get too far yet. First of all, thanks a lot for sharing everything!
I managed to download (hopefully) everything important for the World Machine sessions. Sadly, most of the latest project files are way too complex for my current setup (I have 16gb ram, but I'm running the WM on linux through Windows emulator which is probably why it always crashes while building the project.) Also, there's one File Input which is unnamed and I couldn't deduce what it's supposed to be...
So far I managed to build '30_09_12_New_Hills_Mask_4.tmd' project, which is not as complex as the 'Build_*' projects...
Now I'll have (obviously) much more time to dive into it again, so I hope I'll make some progress with building the 40k successfully.
-
The Minecraft group are not actually interested in moving over to Outerra. Just to clarify.
Yes, I'll be looking at the tmds because the input files into those tmds represent the very latest files. So it would be helpful to work backwards. I'm looking to trace latest base dem, mountains set and 3rd layer. The vectors always resided in the SHPs folders so no problems there.
For the time being the .bts 'Test_*' appear to be the latest base dem.
monks
-
ok, the best candidate for the latest material- that being the base dem and the mountains is the project 270912_New_Mountains_Extents_17_*.gmp. 4 of them numbered A1, A2, B1, B2.
The 3rd terrain layer is the lowlands layer which fills in all of the content not covered by the the other two. It is only added to the terrain in the tmd. Do a search for:
Lowlands_x00_y00.png
You can find all of the tiles in a format useful to GM in the folder 40klowlandtiles.
There are 4 of them. This will give you the latest ones at the end of 2012....so looks like the end of 2012...early 2013 was the last outing of the project.
They are in the folder 301012 which agrees with the date of the gmp above and with the Outerra build file in the following post below. So I've narrowed the final run down to between Sep and the Dec of 2012.
MakeRoomforShadowBuild\HILLS_MASK\40KLowlandTiles\301012
I need to check the heights on the base dem with the heights in the "lops" to determine whether the base dem represents the final for output version from the project. The 'lops' were used to level off to a flat plane at fixed heights, the mountains areas. There are two ways to apply the lops. Either within the project itself or at export. Basically, if they are checked in the Control Centre they will be applied to the exported terrain.
Forget the other projects for now.
There is a txt file called "How_to_rectify_an_image"...which gives you the instructions on how to get data of all kinds into the GM project...and also contains some info using Wintopo.
The last project files are in kinda early 2013 I think. But the terrain in the GM project is extremely unlikely to have changed in that last year.
monks
-
Look what I found...a list of all of the input files into the tmd on Dec 2012. I'll see if that proves to be the latest. It may well be. It looks like I put the south-east terrain in separately as "Africa"...I seem to remember using the Layout for that...not ideal...
OuterraBuild_December12_files.txt
Outerra Build Dec 2012
Tmd: Build271112_LowerDesert
Files:
Base
F:\MEDEM\HILLS_MASK\Base_10K_Coastlines2.hfz
Estuaries
F:\MEDEM\HILLS_MASK\Estuaries_test9.png
Rivers 4 x 20K tiles
F:\MEDEM\HILLS_MASK\40kRivers\40K_rivers_x%.2d_y%.2d.png
Lakes
C:\Medem\OUTERRA_BUILD_PROCESS\Lakes_10K_grey.png
Hills (dummy blank file)
F:\MEDEM\HILLS_MASK\Hills_Dummy.png
Mts Mask
F:\MEDEM\HILLS_MASK\Mountain_Extents_TEST9.png
Mordor Mask
F:\MEDEM\HILLS_MASK\test_grey.png
Wetlands Mask
C:\Medem\OUTERRA_BUILD_PROCESS\20K_Wetlands_grey.png
Real Terrain Lowlands 4 x 20K tiles
F:\MEDEM\HILLS_MASK\40KLowlandTiles\301012\Lowlands_x%.2d_y%.2d.png
Mask for Desert Coastline
F:\MEDEM\HILLS_MASK\DesertCoastlineMask10.png
Desert Terrain (Africa)
F:\MEDEM\HILLS_MASK\Africa_241012\Fix\Africa__x%.2d_y%.2d.hfz
Mountains 40K Mts exported from GM
F:\MEDEM\HILLS_MASK\40K_Mts\251012\Again\40kMtsNoHills_x%.2d_y%.2d.hfz
Mordor Mask Mask for Texture
F:\MEDEM\HILLS_MASK\Mordor_Extents2_20K_grey_2.png
Snow Mask Stops snow on Desert
F:\MEDEM\HILLS_MASK\Africa_SnowMask.png
Africa Mask Texture mask
F:\MEDEM\HILLS_MASK\AfricaMask_20K_2.png
Forest Mask for texturer
F:\MEDEM\HILLS_MASK\Forests.png
monks
-
And here's another...it actually gives the "Africa" settings in the Layout view of WM.
FILE_PATHS.txt
F:\MEDEM\HILLS_MASK\Base_10K_Coastlines2.hfz
F:\MEDEM\HILLS_MASK\Estuaries_test9.png
F:\MEDEM\HILLS_MASK\40kRivers\40K_rivers_x%.2d_y%.2d.png
F:\MEDEM\HILLS_MASK\Lakes_10K_grey_2.png
F:\MEDEM\HILLS_MASK\Hills_Dummy.png
F:\MEDEM\HILLS_MASK\Mountain_Extents_TEST9.png
F:\MEDEM\HILLS_MASK\test_grey.png
F:\MEDEM\HILLS_MASK\20K_Wetlands_grey.png
F:\MEDEM\HILLS_MASK\40KLowlandTiles\301012\Lowlands_x%.2d_y%.2d.png
F:\MEDEM\HILLS_MASK\40K_Mts\251012\Again\40kMtsNoHills_x%.2d_y%.2d.hfz
F:\MEDEM\HILLS_MASK\Africa_241012\Fix\Africa__x%.2d_y%.2d.hfz
F:\MEDEM\HILLS_MASK\DesertCoastlineMask10.png
TEX
F:\MEDEM\HILLS_MASK\Mordor_Extents2_20K_grey_2.png
F:\MEDEM\HILLS_MASK\Africa_SnowMask.png
F:\MEDEM\HILLS_MASK\AfricaMask_20K_2.png
F:\MEDEM\HILLS_MASK\Forests.png
Africa settings:
x = 783.9 Km
y = -0m
Width = 1216.10 Km
Height = 512.05 Km
Width per tile = 608.05 Km
Height pre tile = 512.05 Km
--------------------------
F:\MEDEM\HILLS_MASK\Base_10K_Coastlines2.hfz
F:\MEDEM\HILLS_MASK\Estuaries_test9.png
F:\MEDEM\HILLS_MASK\40kRivers\40K_rivers_x%.2d_y%.2d.png
F:\MEDEM\HILLS_MASK\Lakes_10K_grey_2.png
F:\MEDEM\HILLS_MASK\Hills_Dummy.png
F:\MEDEM\HILLS_MASK\Mountain_Extents_TEST13.png
F:\MEDEM\HILLS_MASK\test_grey.png
F:\MEDEM\HILLS_MASK\20K_Wetlands_grey.png
F:\MEDEM\HILLS_MASK\40KLowlandTiles\301012\Lowlands_x%.2d_y%.2d.png
F:\MEDEM\HILLS_MASK\40K_Mts\251012\Again\Latest\40kMtsNoHills_x%.2d_y%.2d.hfz
F:\MEDEM\HILLS_MASK\Africa_241012\Fix\Africa__x%.2d_y%.2d.hfz
F:\MEDEM\HILLS_MASK\DesertCoastlineMask10.png
TEX
F:\MEDEM\HILLS_MASK\Mordor_Extents2_20K_grey_2.png
F:\MEDEM\HILLS_MASK\Africa_SnowMask.png
F:\MEDEM\HILLS_MASK\AfricaMask_20K_2.png
F:\MEDEM\HILLS_MASK\Forests.png
Africa settings:
x = 783.9 Km
y = -0m
Width = 1216.10 Km
Height = 512.05 Km
Width per tile = 608.05 Km
Height pre tile = 512.05 Km
monks
-
This is the entire build process and sundry other instructions I think...
This 34 KB one not the 30 KB one is the latest...Workflow_for_OUTERRA_BUILD.txt
monks
-
The folder MakeRoomForShadowBuild contains the files which go into the World Machine tmd build.
The World machine input files are in the HillsMask dir.
The parent folder contains other important files. I'm going through the files one by one in my list to see which are there. If I find them all, it might be possible to reconstruct the tmd. Setting up a directory path structure would probably enable opening of it good to go, presuming your machine can open it of course. I'm on 32 GB.
//I found them all. There are 4 terrains. Base dem. Mountains. Lowlands. Africa (south east lower corner). All in tile form. The lowlands went in as a png. The rest are hfz.
So, I'll try and recreate the file structure and then open up the tmd.
I'm also in the middle of creating a tiled gmp with all of the terrain and vector layers in one place.
monks
-
This is the tmd for the last build dates 310113. ;D It has 16 file inputs, many are tiled sets. I rebuilt the whole dir structure but it didn't load any files. Tmd loaded fine. So I'll try rebuild it manually. I reckon all of the files going to the Mincraft peeps could just as easily be exported from the tmd network as from Global Mapper.
I might not do things the same way this time around. The tmd might be different. So much has changed with WM anyway in 7 years.
https://my.pcloud.com/publink/show?code=XZX6SMkZHhoyPaP0QABXDqTV7xjE684cUUQX
monks