Doc automation (oneapi-src#637)

* update

* update

* misc updates'

* updates

mis

* Update CODESAMPLESLIST.md

* format issues

* Update README.md

* Update README.md

* misc updates

* Update README.md

* update for readability

Author: JoeOster

.repo-tools/Docs_Automation/content.json

@@ -3,7 +3,7 @@
 
   "mdContribute": "## Contribute\n\nSee [CONTRIBUTING](https://github.com/oneapi-src/oneAPI-samples/blob/master/CONTRIBUTING.md) for more itargetedDevices.ormation.\n\n",
 
-  "mdIntro1":"The oneAPI samples repository provides code samples for Intel oneAPI toolkits.<br><br>We recommend checking out a specific stable release version of the repository to [View available stable releases](https//github.com/oneapi-src/oneAPI-samples/tags). ",
+  "mdIntro1":"The oneAPI samples repository provides code samples for Intel oneAPI toolkits.<br><br>We recommend checking out a specific stable release version of the repository. [View available stable releases](https//github.com/oneapi-src/oneAPI-samples/tags). ",
   "mdIntro2":"The latest versions (",
   "mdIntro2.1":") of code samples on the master branch are not guaranteed to be stable.",
   "mdIntro3":"The oneAPI sample repository is organized as follows:",
@@ -12,23 +12,23 @@
   "mdIntro3.3":"* [Libraries](https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries)\n  * [oneCCl](https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries/oneCCL)\n  * [oneDAL](https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries/oneDAL)\n  * [oneDNN](https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries/oneDNN)\n  * [oneDPL](https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries/oneDPL)\n  * [oneMKL](https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries/oneMKL)\n  * [oneTBB](https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries/oneTBB)\n  * [oneVPL](https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries/oneVPL)\n",
   "mdIntro3.4":"* [Publications](https://github.com/oneapi-src/oneAPI-samples/Publications/)\n  * [Data Parallel C++](https://github.com/oneapi-src/oneAPI-samples/Publications/Data_Parallel_C%2B%2B)\n",
   "mdIntro3.5":"* [Tools](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/)\n  * [Advisor](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/Advisor)\n  * [Application Debugger](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/ApplicationDebugger)\n  * [Benchmark](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/Benchmark)\n  * [IoT Connections Tools](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/IoTConnectionsTools)\n  * [Migration](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/Migration)\n  * [Socwatch](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/Socwatch)\n  * [Trace](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/Trace)\n  * [UEFI debug](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/UEFI%20debug)\n  * [VTune Profiler](https://github.com/oneapi-src/oneAPI-samples/tree/master/Tools/VTuneProfiler)\n)",  "mdIntro4":"To view an alphabetized list of all samples with descriptions by:", 
-  "mdIntro4.1":"- Samples in Alphabetical order w/ device target](https://github.com/oneapi-src/CODESAMPLESLIST.md/)",
-  "mdIntro4.2":"- Samples by [Change History](https://github.com/oneapi-src/oneAPI-samples/CHANGELOGS.md)",
+  "mdIntro4":"## Additional Resources\n- Samples in [Alphabetical order w/ device target](https://github.com/oneapi-src/CODESAMPLESLIST.md/)\n- Samples by [Change History](https://github.com/oneapi-src/oneAPI-samples/CHANGELOGS.md)",
+  "mdIntro5":"\n\n## Known Issues or Limitations\n\n### On Windows Platform\n\n",
   "mdIntro5.1":"- If you are using Visual Studio 2019, Visual Studio 2019 version 16.4.0 or newer is required.",
   "mdIntro5.2":"- Windows support for the FPGA code samples is limited to the FPGA emulator and optimization reports. Compile targets for FPGA hardware are provided on Linux only. See any FPGA code sample for more details.",
   "mdIntro5.3":"- If you encounter a compilation error when building a sample program, such as the example error below, the directory path of the sample may be too long. The workaround is to move the sample to a temp directory.",
-  "mdIntro5.4":"- Example error: *Error MSB6003 The specified task executable dpcpp.exe could not be run .......",
-  "mdIntro6":"See [CONTRIBUTING wiki](https://github.com/oneapi-src/oneAPI-samples/blob/master/CONTRIBUTING.md) for more information.",
+  "mdIntro5.4":"    - Example error: *Error MSB6003 The specified task executable dpcpp.exe could not be run .......",
+  "mdIntro6":"\n\n## Contributing\n\nSee [CONTRIBUTING wiki](https://github.com/oneapi-src/oneAPI-samples/blob/master/CONTRIBUTING.md) for more information.",
   "mdIntro7":"\n\n## New Code Samples\n\n|Version Introduced   |Sample Name|Description|\n |-----------------------|-------------------------------------------|---------------|\n",
 
-  "mdDeviceTargetIntrop1": "## oneAPI Code Samples\n\n",
-  "mdDeviceTargetIntrop2":  "|Code Sample Name|Supported Intel&reg; Architecture(s)|Description|\n|-----------------------|----------------------|---------------------|\n",
+  "mdCodeSamplesListIntrop1": "## oneAPI Code Samples\n\nThis Page is a complete list of onePI Code Samples, sorted by Alphabetical order",
+  "mdCodeSamplesListIntrop2":  "|Code Sample Name|Supported Intel&reg; Architecture(s)|Description|\n|-----------------------|----------------------|---------------------|\n",
 
-  "mdDeviceTargetFooter":  "Report Generated on:  ",
+  "mdCodeSamplesListFooter":  "Report Generated on:  ",
 
   "mdVersion": "## Version List of all samples\n\n sorted as found in directories\n\n|Code Sample/URL |Version|\n|-----------------------|------------------------|\n",
 
-  "mdChangeLogHeaderp1":"## oneAPI Code Sample Change log\n\nThis document shows the history of when a specific sample was introduced to the oneAPI ecosystem of Code Samples.<br>Current Samples Count: ",
+  "mdChangeLogHeaderp1":"## oneAPI Code Sample Change log\n\nThis document shows the history of when a specific sample was introduced to the oneAPI ecosystem of Code Samples.",
   "mdChangeLogHeaderp2":"\n| Version|Code Sample Name|description|\n|-----------------------|------------------|-------------------------|\n"
 
 }

.repo-tools/Docs_Automation/create_docs.py

@@ -1,20 +1,18 @@
+from datetime import date
+from collections import OrderedDict
 from json.decoder import JSONDecodeError
 from pathlib import Path
-from datetime import date
 import os
 import json
-from collections import OrderedDict
 today = date.today()
+d = today.strftime("%B %d, %Y")
 currentVersion = "2021.4.0"
 fileName = "sample.json"
-fDeviceTargets = "CODESAMPLESLIST.md"
+fCodeSamplesLists = "CODESAMPLESLIST.md"
 fChangeLogs = "CHANGELOGS.md"
 freadme = "README.md"
 fguidVer = ".repo-tools\Docs_Automation\guids.json"
-absolute_path = os.getcwd()
-pathLength = len(absolute_path)   #used to modify local path length, so only tree structure is left for crafting the url
 oneAPIURL = 'https://github.com/oneapi-src/oneAPI-samples/tree/master'
-d = today.strftime("%B %d, %Y")
 count = 0
 
 def checkFileExists(checkFile):
@@ -23,37 +21,6 @@ def checkFileExists(checkFile):
     else:
         print("The " + checkFile + " file does not exist")
 
-def createHeaders(dataContent):
-    nf = open(fDeviceTargets,"w+")
-    nf.write(dataContent['mdDeviceTargetIntrop1'] + dataContent['mdDeviceTargetIntrop2'])
-    nf.close()
-    nf = open(fChangeLogs,"w+")
-    nf.write(dataContent['mdChangeLogHeaderp1'])
-    nf.close()
-    nf = open(freadme,"w+")
-    nf.write("## Introduction\n\n")
-    nf.write(dataContent['mdIntro1'] + "\n" +dataContent['mdIntro2'] + currentVersion + dataContent['mdIntro2.1'] + "\n ### Sample Details\n\n")
-    nf.write(dataContent['mdIntro3'] + dataContent['mdIntro3.1']+dataContent['mdIntro3.2'] + dataContent['mdIntro3.3'] + dataContent['mdIntro3.4'] + dataContent['mdIntro3.5'])
-    nf.write("\n" +dataContent['mdIntro4'] + "\n" +dataContent['mdIntro4.1'] + "\n" +dataContent['mdIntro4.2'])
-    nf.write("\n\n### On Windows Platform\n\n" + dataContent['mdIntro5.1'] + "\n" + dataContent['mdIntro5.2'] + "\n" + dataContent['mdIntro5.3'] + "\n" + dataContent['mdIntro5.4'])
-    nf.write("\n\n## Known Issues or Limitations\n\n## Contributing\n\n" + dataContent['mdIntro6'] + "\n\n" + dataContent['mdIntro7'])
-    nf.close()
-
-def createFooters(count):
-    #setting up Device Targets
-    nf = open(fDeviceTargets,"a")
-    nf.write("Total Samples: " + str(count)+ "\n\n")
-    nf.write(str(dataContent['mdDeviceTargetFooter']) + d)
-    nf.write("\nThis doc was created on: " + d)
-    nf.close()
-    nf = open(freadme,"a+")
-    nf.write(dataContent['mdLicense'])
-    nf.write("\nThis doc was created on: " + d)
-    nf.close()
-    nf = open(fDeviceTargets,"a+")
-    nf.write("\nThis doc was created on: " + d)
-    nf.close()
-    
 def openJson(jsonFile):                 #creating a dictionary
     jsonData = open(jsonFile)           #open the json file
     try: 
@@ -62,113 +29,127 @@ def openJson(jsonFile):                 #creating a dictionary
         print(str(e)+': ' + jsonFile)
     return data
 
-def readContent():                                  #readin in strings for use in document creation
+def readContent():                      #reading in strings for use in document creation
     jsonFile = '.repo-tools\Docs_Automation\content.json'
     dataContent = openJson(jsonFile)
     return dataContent
-
-def addVersion(dict_main, dict_version):# After walking thu directories we need to add version to dict_main
-    for key in dict_version.keys():
-        try:
-            if (key in dict_main):
-                dict_tmp=dict_version[key]  #copy to dict_tmp if true 
-                ver=dict_tmp['ver']
-                dict_main[key]['ver']=ver
-
-        except KeyError as e: #not working
-            print(str(e) + ":No Match for guid: ")
 
-def createChangeLog(count): #sorted but does not include version
-    temp = dict_main.items()
-    sorted_items = sorted(temp, key=lambda key_value: key_value[1]["name"], reverse=False) # sorts by name
-    sorted_by_name = OrderedDict(sorted_items)
-
-    temp=dict_version.items()
-    sorted_items = sorted(temp, key=lambda key_value: key_value[1]["ver"], reverse=True) # sorts by ver
-    sorted_by_ver = OrderedDict(sorted_items)
-
-    nf = open(fChangeLogs,"a+")
-    nf.write(str(count) + dataContent['mdChangeLogHeaderp2'])
-    for key in sorted_by_name.keys():
+def createChangeLog(count,sorted_by_name,sorted_by_ver): #sorted but does not include version
+    nf = open(fChangeLogs,"w+")
+    nf.write(dataContent['mdChangeLogHeaderp1'])
+    nf.write(dataContent['mdChangeLogHeaderp2'])
+  
+    for key in sorted_by_ver.keys():
         description= str(sorted_by_name[key]['description']) 
         url= sorted_by_name[key]['url']
         ver=sorted_by_ver[key]['ver']
         name=sorted_by_name[key]['name']
         cat=str(sorted_by_name[key]['categories'])
 
+        # Due to name issues, we need to fix the DPC** books chapter namesas its found and put it into the doc
         if (cat=="['Toolkit/Publication: Data Parallel C++']"):
-            description=description.replace('*','<br>')
-            name ="Pub: Data Parallel C++:](https://www.apress.com/9781484255735)" + "<br><br>[" + name
+            name ="Pub: Data Parallel C++:](https://www.apress.com/9781484255735)<br>[" + name
+            description=description.replace('*','')
+            description=description.replace('fig_','<br>- Fig_')
+            description="Collection of Code samples for the chapter"+description
+        
         nf.write("|" + ver + "|[" + name+ "](" + url + ")|" + description + "|\n") 
+    nf.write("Total Samples: " + str(count)+ "\n\n")
+    nf.write(str(dataContent['mdCodeSamplesListFooter']) + d)
     nf.close()
+    print("Change Log has been created")
 
-def createTtargetedDevices():
+def createCodeSamplesList():
     temp = dict_main.items()
     sorted_items = sorted(temp, key=lambda key_value: key_value[1]["name"], reverse=False) # sorts by name
     sorted_by_name = OrderedDict(sorted_items)
     temp=sorted_by_name.items()
-    nf = open(fDeviceTargets,"a+")
-
+    nf = open(fCodeSamplesLists,"w+")
+    nf.write(dataContent['mdCodeSamplesListIntrop1'] + "\n\n" + dataContent['mdCodeSamplesListIntrop2'])
     for key in sorted_by_name.keys():
         description= str(sorted_by_name[key]['description']) 
         url= sorted_by_name[key]['url']
         name=sorted_by_name[key]['name']
         target= str(sorted_by_name[key]['targetDevice'])
         cat=str(sorted_by_name[key]['categories'])
         if (cat=="""['Toolkit/Publication: Data Parallel C++']"""):
-            description=description.replace('*','<br>')
-            name ="Pub: Data Parallel C++:](https://www.apress.com/9781484255735)" + "<br><br>[" + name
+            description=description.replace('*','')
+            description=description.replace('fig_','<br>- Fig_')
+            description="Collection of Code samples for the chapter"+description
+            name ="Pub: Data Parallel C++:](https://www.apress.com/9781484255735)<br>[" + name
         nf.write("|[" + name+ "](" + url + ")|" + target + "|" + description + "|\n") 
+    
+    nf.write("Total Samples: " + str(count)+ "\n\n")
+    nf.write(str(dataContent['mdCodeSamplesListFooter']) + d)
+    nf.write("Total Samples: " + str(count)+ "\n\n")
     nf.close()
+    print("Code Samples List has been created")
 
-def createReadme():
-    temp = dict_main.items()
-    sorted_items = sorted(temp, key=lambda key_value: key_value[1]["name"], reverse=False) # sorts by name
-    sorted_by_name = OrderedDict(sorted_items)
-    nf = open(freadme,"a+")
-    
+def createReadme(sorted_by_name, sorted_by_ver):
+    nf = open(freadme,"w+")
+    nf.write("## Introduction\n\n")
+    nf.write(dataContent['mdIntro1'] + "\n" +dataContent['mdIntro2'] + currentVersion + dataContent['mdIntro2.1'] + "\n ### Sample Details\n\n")
+    nf.write(dataContent['mdIntro3'] + dataContent['mdIntro3.1']+dataContent['mdIntro3.2'] + dataContent['mdIntro3.3'] + dataContent['mdIntro3.4'] + dataContent['mdIntro3.5'])
+    nf.write(dataContent['mdIntro5'] + dataContent['mdIntro5.1'] + "\n" + dataContent['mdIntro5.2'] + "\n" + dataContent['mdIntro5.3'] + "\n" + dataContent['mdIntro5.4'])
+    nf.write("\n\n" +dataContent['mdIntro4'])
+    nf.write(dataContent['mdIntro6'] + "\n\n" + dataContent['mdIntro7'])
+        
     for key in sorted_by_name.keys():
         description= str(sorted_by_name[key]['description']) 
         url= sorted_by_name[key]['url']
-        ver=sorted_by_name[key]['ver']
+        ver=sorted_by_ver[key]['ver']
         name=sorted_by_name[key]['name']
         cat=str(sorted_by_name[key]['categories'])
         if (cat=="""['Toolkit/Publication: Data Parallel C++']"""):
-            name ="Pub: Data Parallel C++:](https://www.apress.com/9781484255735)" + "\n\n[" + name
-            description=description.replace('*','<br>')
+            name ="Pub: Data Parallel C++:](https://www.apress.com/9781484255735)<br>[" + name
+            description=description.replace('*','')
+            description=description.replace('fig_','<br>- Fig_')
+            description="Collection of Code samples for the chapter"+description
         if (ver==currentVersion):
             nf.write("|" + ver + "|[" + name+ "](" + url + ")|" + description + "|\n") 
+    nf.write("\nTotal Samples: " + str(count)+ "\n\n")
+    nf.write(dataContent['mdLicense'])
+    nf.write("\n\n")
+    nf.write(str(dataContent['mdCodeSamplesListFooter']) + d)
     nf.close()
-   
+    print("Readme has been created")
+
 #main
-checkFileExists(fDeviceTargets)     #Cleaning up from previous run
+checkFileExists(fCodeSamplesLists)     #Cleaning up from previous run
 checkFileExists(fChangeLogs)        #Cleaning up from previous run
 checkFileExists(freadme)            #Cleaning up from previous run
 dataContent = readContent()         #read json for data used in creating document header and footers
-createHeaders(dataContent)          #create headers for the various documents being generated
 
 dict_main={}                        
 dict_version = openJson(fguidVer)
 
-for subdir, dirs, files in os.walk('..\\'):
+for subdir, dirs, files in os.walk('..\\'):  # walk through samples repo looking for samples.json, if found add data to dict_main
     for file in files:
         if (file == fileName):
             f = os.path.join(subdir, file)
+            data = openJson(f) 
+            dict_main[data['guid']]=data   
             # build url
             fp = os.path.join(subdir)
             fp = fp.replace('\\','/')                 #char replace \ for /
             fullURL=oneAPIURL+(str(fp)[17:])  # removed first 17 characters of the path, which is always "../oneAPI-samples/"
             #end build url
-            data = openJson(f) 
-            dict_main[data['guid']]=data   
             dict_main[data['guid']]['url'] = fullURL
             count = count+1
+
+temp = dict_main.items()
+sorted_by_name = OrderedDict(sorted(temp, key=lambda key_value: key_value[1]["name"], reverse=False))
+
+temp=dict_version.items()
+sorted_by_ver = OrderedDict(sorted(temp, key=lambda key_value: key_value[1]["ver"], reverse=True))
         # Future - add a search for license file and if none, show a warning
         # Future - if no sample.json is present then show a warning
         # future - if no readme.md is present then show a warning
-addVersion(dict_main,dict_version)
-createChangeLog(count)
-createTtargetedDevices()
-createReadme()
-createFooters(count)
+        # future - Check dict_main vs dict_version for guid present if not then need to add
+        # furure - check dict_version vs dict_main for guid present if not then need to allow if new sample hasnt been uploaded
+
+createChangeLog(count,sorted_by_name,sorted_by_ver)
+createCodeSamplesList()
+createReadme(sorted_by_name,sorted_by_ver)
+
 print("Finished")

.repo-tools/Docs_Automation/guids.json

@@ -689,12 +689,6 @@
         "name": "Particle Diffusion",
         "notes": "-"
     },
-    "668A3FF9-12C3-4559-8AF4-54B6792122D3": {
-        "guid": "668A3FF9-12C3-4559-8AF4-54B6792122D3",
-        "ver": "2021.1.Gold",
-        "name": "System Debug",
-        "notes": "-"
-    },
     "314B4C1D-07A0-4527-A9FA-9F87ED9622D7": {
         "guid": "314B4C1D-07A0-4527-A9FA-9F87ED9622D7",
         "ver": "2021.2.1",
@@ -713,13 +707,6 @@
         "name": "Fourier Correlation",
         "notes": "-"
     },
-    "7909FAE1-D3D4-4E97-A963-14A884F33495": {
-        "guid": "7909FAE1-D3D4-4E97-A963-14A884F33495",
-        "ver": "2021.2.1",
-        "name": "Max Concurrency",
-        "notes": "Removed in 2021.4",
-        "removed": "x"
-    },
     "8401B5F5-6F09-4727-A080-684C57104CAF": {
         "guid": "8401B5F5-6F09-4727-A080-684C57104CAF",
         "ver": "2021.2.1",
@@ -919,11 +906,6 @@
         "ver": "2021.1.Gold",
         "name": "Tutorials"
     },
-    "dc8c1c92-1c2a-4edf-abbd-6c35dfab256a": {
-        "guid": "dc8c1c92-1c2a-4edf-abbd-6c35dfab256a",
-        "ver": "2021.4.0",
-        "name": "VisualComputing tutorial"
-    },
     "748eb906-309f-469b-95c9-5258006c7587": {
         "guid": "748eb906-309f-469b-95c9-5258006c7587",
         "ver": "2021.1.Gold",