All of this happens in the Bullets & Numbering dialog box, shown below. You will definitely want to use paragraph styles for this. My first one is called Chapter title. You will need to begin by changing the List Type to Numbers for all of the levels, and you must both name the List and use the same named list for all Levels. You do this by selecting the List > New List.
In the Adobe InDesign document I’m working on here we actually have two master pages, an ‘A’ master and a ‘B’ master so I’m going to select both the left and right hand page frames with the special character in on my ‘A’ master and go to Edit>Copy. I’m then going to double click on my ‘B’ master page and go to Edit>Paste in place. this will then copy the special character and position it in exactly the same place as it appears on my ‘A’ master. numbering in publisher