handling description markup/semantics for SQLite3 "output"?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

handling description markup/semantics for SQLite3 "output"?

Travis Everett
All,

I'm resuming work on refining the experimental SQLite3 output. I've run into a pretty obvious quandary that I was blissfully ignorant of:

Before output-format-specific conversion, it looks like detailed descriptions can include (at least) a mix of HTML and raw text that still contains doxygen commands (I haven't gone fishing, but my guess is that markdown gets translated down to HTML, and output translators are responsible for the rest).

When it's being served by an SQL database, it doesn't seem like there's an "obvious" or "expected" format for presentational/semantic markup embedded in the detailed description. I was a bit surprised to find plaintext Doxygen commands present at this stage. 

I'm curious if anyone has thoughts on how to handle these. The best three paths forward seem to be:

- Saving it as XML. I'm not keen on this since it significantly raises the bar on using the sql output, and would require either updating all of the descriptions in place, or parsing them into a more useful form on each use.
- Saving an opinionated plaintext translation. This would retain simple usability and avoid the update-all-descriptions or constantly-reparse issues, as long as consumers can live with the format ;)
- Some active, user-configurable translation method (probably translate to XML and then call a user-defined translator script as with input filters. *not sure how complex this would be or where to start; may be getting out of my depth.

Thanks for any direction,
Travis

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: handling description markup/semantics for SQLite3 "output"?

Negreanu Marius
Hi Travis,

The format of the description is dictated by the client's needs. Or in other words, where that description is used.

I'm using it from python scripts which makes it easy to parse the current plain text.

This brings me to the question : why do you need another format, say (A), when you'll already have to parse the plaintext format to extract the semantic needed by (A) ?

One reason for adding (A) is "it's easier for the client to use/parse", which brings me again, to the first part : what client do you have ?

Regards


On Jul 10, 2017 3:21 AM, "Travis Everett" <[hidden email]> wrote:
All,

I'm resuming work on refining the experimental SQLite3 output. I've run into a pretty obvious quandary that I was blissfully ignorant of:

Before output-format-specific conversion, it looks like detailed descriptions can include (at least) a mix of HTML and raw text that still contains doxygen commands (I haven't gone fishing, but my guess is that markdown gets translated down to HTML, and output translators are responsible for the rest).

When it's being served by an SQL database, it doesn't seem like there's an "obvious" or "expected" format for presentational/semantic markup embedded in the detailed description. I was a bit surprised to find plaintext Doxygen commands present at this stage. 

I'm curious if anyone has thoughts on how to handle these. The best three paths forward seem to be:

- Saving it as XML. I'm not keen on this since it significantly raises the bar on using the sql output, and would require either updating all of the descriptions in place, or parsing them into a more useful form on each use.
- Saving an opinionated plaintext translation. This would retain simple usability and avoid the update-all-descriptions or constantly-reparse issues, as long as consumers can live with the format ;)
- Some active, user-configurable translation method (probably translate to XML and then call a user-defined translator script as with input filters. *not sure how complex this would be or where to start; may be getting out of my depth.

Thanks for any direction,
Travis

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: handling description markup/semantics for SQLite3 "output"?

Travis Everett
Hi again Adrian, 

I agree with what you've said, but I think that only holds when the detaileddescription database field holds a single format.

An example of the input/output I'm working with might be more useful than my initial description: https://gist.github.com/abathur/b3d3d25885303eb216e0ffa2d2604034

In this example, you can see that the string going into the database contains three different formats: an HTML header, plaintext markup, and Doxygen commands. 

Context: `generateSqlite3ForPage` is one of the parts of sqlite3gen that I'm refining as I work on a broader project that generates Doxygen-ready source files and markdown pages from a large Sphinx/RST corpus. 

Cheers,
Travis

On Sun, Jul 9, 2017 at 11:48 PM, Adrian M Negreanu <[hidden email]> wrote:
Hi Travis,

The format of the description is dictated by the client's needs. Or in other words, where that description is used.

I'm using it from python scripts which makes it easy to parse the current plain text.

This brings me to the question : why do you need another format, say (A), when you'll already have to parse the plaintext format to extract the semantic needed by (A) ?

One reason for adding (A) is "it's easier for the client to use/parse", which brings me again, to the first part : what client do you have ?

Regards


On Jul 10, 2017 3:21 AM, "Travis Everett" <[hidden email]> wrote:
All,

I'm resuming work on refining the experimental SQLite3 output. I've run into a pretty obvious quandary that I was blissfully ignorant of:

Before output-format-specific conversion, it looks like detailed descriptions can include (at least) a mix of HTML and raw text that still contains doxygen commands (I haven't gone fishing, but my guess is that markdown gets translated down to HTML, and output translators are responsible for the rest).

When it's being served by an SQL database, it doesn't seem like there's an "obvious" or "expected" format for presentational/semantic markup embedded in the detailed description. I was a bit surprised to find plaintext Doxygen commands present at this stage. 

I'm curious if anyone has thoughts on how to handle these. The best three paths forward seem to be:

- Saving it as XML. I'm not keen on this since it significantly raises the bar on using the sql output, and would require either updating all of the descriptions in place, or parsing them into a more useful form on each use.
- Saving an opinionated plaintext translation. This would retain simple usability and avoid the update-all-descriptions or constantly-reparse issues, as long as consumers can live with the format ;)
- Some active, user-configurable translation method (probably translate to XML and then call a user-defined translator script as with input filters. *not sure how complex this would be or where to start; may be getting out of my depth.

Thanks for any direction,
Travis

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: handling description markup/semantics for SQLite3 "output"?

Negreanu Marius
Hi,
 I think, from the gist you posted, that your use case needs to extend the current schema so that the elements you see in the xml will also be found in sqlite3.

So you'll need a `page` table and add a generateSqlite3ForPage that fills in that table.
Tbis way the semantic you mentioned previously would be provided by the new table/attribute.

On Jul 10, 2017 18:57, "Travis Everett" <[hidden email]> wrote:
Hi again Adrian, 

I agree with what you've said, but I think that only holds when the detaileddescription database field holds a single format.

An example of the input/output I'm working with might be more useful than my initial description: https://gist.github.com/abathur/b3d3d25885303eb216e0ffa2d2604034

In this example, you can see that the string going into the database contains three different formats: an HTML header, plaintext markup, and Doxygen commands. 

Context: `generateSqlite3ForPage` is one of the parts of sqlite3gen that I'm refining as I work on a broader project that generates Doxygen-ready source files and markdown pages from a large Sphinx/RST corpus. 

Cheers,
Travis

On Sun, Jul 9, 2017 at 11:48 PM, Adrian M Negreanu <[hidden email]> wrote:
Hi Travis,

The format of the description is dictated by the client's needs. Or in other words, where that description is used.

I'm using it from python scripts which makes it easy to parse the current plain text.

This brings me to the question : why do you need another format, say (A), when you'll already have to parse the plaintext format to extract the semantic needed by (A) ?

One reason for adding (A) is "it's easier for the client to use/parse", which brings me again, to the first part : what client do you have ?

Regards


On Jul 10, 2017 3:21 AM, "Travis Everett" <[hidden email]> wrote:
All,

I'm resuming work on refining the experimental SQLite3 output. I've run into a pretty obvious quandary that I was blissfully ignorant of:

Before output-format-specific conversion, it looks like detailed descriptions can include (at least) a mix of HTML and raw text that still contains doxygen commands (I haven't gone fishing, but my guess is that markdown gets translated down to HTML, and output translators are responsible for the rest).

When it's being served by an SQL database, it doesn't seem like there's an "obvious" or "expected" format for presentational/semantic markup embedded in the detailed description. I was a bit surprised to find plaintext Doxygen commands present at this stage. 

I'm curious if anyone has thoughts on how to handle these. The best three paths forward seem to be:

- Saving it as XML. I'm not keen on this since it significantly raises the bar on using the sql output, and would require either updating all of the descriptions in place, or parsing them into a more useful form on each use.
- Saving an opinionated plaintext translation. This would retain simple usability and avoid the update-all-descriptions or constantly-reparse issues, as long as consumers can live with the format ;)
- Some active, user-configurable translation method (probably translate to XML and then call a user-defined translator script as with input filters. *not sure how complex this would be or where to start; may be getting out of my depth.

Thanks for any direction,
Travis

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: handling description markup/semantics for SQLite3 "output"?

Travis Everett
I think maybe we're talking past each other :)

I'm already building out the `generateSqlite3ForPage` work-in-progress stub you created, and pages currently fit within the existing compounddef SQL table just as well as they fit within the compounddef XML schema.

My question is whether/how the detaileddescription can be saved in a single format, so that it's not in at least 3 different formats. 

I noticed this problem while building out the "page" type (and used it as a succinct example of the issue), but it already exists for any fields that can hold arbitrary nested input (such as complex detaileddescriptions on memberdefs). I've added a 6th file at the end of the gist (https://gist.github.com/abathur/b3d3d25885303eb216e0ffa2d2604034containing the database row for one of our more complex member definitions, demonstrating this issue. This problem will apply to all additional compound types as I build them out. 

Hope this clarifies,
T

On Mon, Jul 10, 2017 at 11:05 AM, Adrian M Negreanu <[hidden email]> wrote:
Hi,
 I think, from the gist you posted, that your use case needs to extend the current schema so that the elements you see in the xml will also be found in sqlite3.

So you'll need a `page` table and add a generateSqlite3ForPage that fills in that table.
Tbis way the semantic you mentioned previously would be provided by the new table/attribute.

On Jul 10, 2017 18:57, "Travis Everett" <[hidden email]> wrote:
Hi again Adrian, 

I agree with what you've said, but I think that only holds when the detaileddescription database field holds a single format.

An example of the input/output I'm working with might be more useful than my initial description: https://gist.github.com/abathur/b3d3d25885303eb216e0ffa2d2604034

In this example, you can see that the string going into the database contains three different formats: an HTML header, plaintext markup, and Doxygen commands. 

Context: `generateSqlite3ForPage` is one of the parts of sqlite3gen that I'm refining as I work on a broader project that generates Doxygen-ready source files and markdown pages from a large Sphinx/RST corpus. 

Cheers,
Travis

On Sun, Jul 9, 2017 at 11:48 PM, Adrian M Negreanu <[hidden email]> wrote:
Hi Travis,

The format of the description is dictated by the client's needs. Or in other words, where that description is used.

I'm using it from python scripts which makes it easy to parse the current plain text.

This brings me to the question : why do you need another format, say (A), when you'll already have to parse the plaintext format to extract the semantic needed by (A) ?

One reason for adding (A) is "it's easier for the client to use/parse", which brings me again, to the first part : what client do you have ?

Regards


On Jul 10, 2017 3:21 AM, "Travis Everett" <[hidden email]> wrote:
All,

I'm resuming work on refining the experimental SQLite3 output. I've run into a pretty obvious quandary that I was blissfully ignorant of:

Before output-format-specific conversion, it looks like detailed descriptions can include (at least) a mix of HTML and raw text that still contains doxygen commands (I haven't gone fishing, but my guess is that markdown gets translated down to HTML, and output translators are responsible for the rest).

When it's being served by an SQL database, it doesn't seem like there's an "obvious" or "expected" format for presentational/semantic markup embedded in the detailed description. I was a bit surprised to find plaintext Doxygen commands present at this stage. 

I'm curious if anyone has thoughts on how to handle these. The best three paths forward seem to be:

- Saving it as XML. I'm not keen on this since it significantly raises the bar on using the sql output, and would require either updating all of the descriptions in place, or parsing them into a more useful form on each use.
- Saving an opinionated plaintext translation. This would retain simple usability and avoid the update-all-descriptions or constantly-reparse issues, as long as consumers can live with the format ;)
- Some active, user-configurable translation method (probably translate to XML and then call a user-defined translator script as with input filters. *not sure how complex this would be or where to start; may be getting out of my depth.

Thanks for any direction,
Travis

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Doxygen-develop mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/doxygen-develop
Loading...